2023-03-21 11:55:27,683 DEBUG [main] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/bb9d57e0-7c01-5166-0245-f9bc94a789fd 2023-03-21 11:55:27,695 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.TestAcidGuaranteesWithEagerPolicy timeout: 13 mins 2023-03-21 11:55:27,707 INFO [Time-limited test] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=1, rsPorts=, rsClass=null, numDataNodes=1, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-03-21 11:55:27,708 INFO [Time-limited test] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/bb9d57e0-7c01-5166-0245-f9bc94a789fd/cluster_9b0830d2-29fd-b47c-c103-f4aba275458e, deleteOnExit=true 2023-03-21 11:55:27,708 INFO [Time-limited test] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-03-21 11:55:27,708 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/bb9d57e0-7c01-5166-0245-f9bc94a789fd/test.cache.data in system properties and HBase conf 2023-03-21 11:55:27,709 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/bb9d57e0-7c01-5166-0245-f9bc94a789fd/hadoop.tmp.dir in system properties and HBase conf 2023-03-21 11:55:27,709 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/bb9d57e0-7c01-5166-0245-f9bc94a789fd/hadoop.log.dir in system properties and HBase conf 2023-03-21 11:55:27,710 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/bb9d57e0-7c01-5166-0245-f9bc94a789fd/mapreduce.cluster.local.dir in system properties and HBase conf 2023-03-21 11:55:27,710 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/bb9d57e0-7c01-5166-0245-f9bc94a789fd/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-03-21 11:55:27,710 INFO [Time-limited test] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-03-21 11:55:27,804 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2023-03-21 11:55:28,174 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-03-21 11:55:28,177 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/bb9d57e0-7c01-5166-0245-f9bc94a789fd/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-03-21 11:55:28,178 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/bb9d57e0-7c01-5166-0245-f9bc94a789fd/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-03-21 11:55:28,178 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/bb9d57e0-7c01-5166-0245-f9bc94a789fd/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-03-21 11:55:28,178 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/bb9d57e0-7c01-5166-0245-f9bc94a789fd/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-03-21 11:55:28,179 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/bb9d57e0-7c01-5166-0245-f9bc94a789fd/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-03-21 11:55:28,179 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/bb9d57e0-7c01-5166-0245-f9bc94a789fd/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-03-21 11:55:28,179 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/bb9d57e0-7c01-5166-0245-f9bc94a789fd/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-03-21 11:55:28,179 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/bb9d57e0-7c01-5166-0245-f9bc94a789fd/dfs.journalnode.edits.dir in system properties and HBase conf 2023-03-21 11:55:28,180 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/bb9d57e0-7c01-5166-0245-f9bc94a789fd/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-03-21 11:55:28,180 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/bb9d57e0-7c01-5166-0245-f9bc94a789fd/nfs.dump.dir in system properties and HBase conf 2023-03-21 11:55:28,180 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/bb9d57e0-7c01-5166-0245-f9bc94a789fd/java.io.tmpdir in system properties and HBase conf 2023-03-21 11:55:28,180 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/bb9d57e0-7c01-5166-0245-f9bc94a789fd/dfs.journalnode.edits.dir in system properties and HBase conf 2023-03-21 11:55:28,181 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/bb9d57e0-7c01-5166-0245-f9bc94a789fd/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-03-21 11:55:28,181 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/bb9d57e0-7c01-5166-0245-f9bc94a789fd/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-03-21 11:55:28,589 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-03-21 11:55:28,593 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-03-21 11:55:28,818 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2023-03-21 11:55:28,962 INFO [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Reload4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2023-03-21 11:55:28,976 WARN [Time-limited test] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-03-21 11:55:29,008 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2023-03-21 11:55:29,034 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/bb9d57e0-7c01-5166-0245-f9bc94a789fd/java.io.tmpdir/Jetty_localhost_localdomain_36975_hdfs____.kpamxe/webapp 2023-03-21 11:55:29,142 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost.localdomain:36975 2023-03-21 11:55:29,170 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-03-21 11:55:29,170 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-03-21 11:55:29,519 WARN [Listener at localhost.localdomain/35467] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-03-21 11:55:29,601 WARN [Listener at localhost.localdomain/35467] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-03-21 11:55:29,616 WARN [Listener at localhost.localdomain/35467] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-03-21 11:55:29,622 INFO [Listener at localhost.localdomain/35467] log.Slf4jLog(67): jetty-6.1.26 2023-03-21 11:55:29,627 INFO [Listener at localhost.localdomain/35467] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/bb9d57e0-7c01-5166-0245-f9bc94a789fd/java.io.tmpdir/Jetty_localhost_43277_datanode____g7lxy2/webapp 2023-03-21 11:55:29,706 INFO [Listener at localhost.localdomain/35467] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:43277 2023-03-21 11:55:29,978 WARN [Listener at localhost.localdomain/37475] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-03-21 11:55:30,291 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x67811143bcd8a2f9: Processing first storage report for DS-6ec069ee-2435-4585-a074-39f4d5d52ec1 from datanode 13b4a3a9-8383-46bc-9d4a-34a9f4509b76 2023-03-21 11:55:30,292 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x67811143bcd8a2f9: from storage DS-6ec069ee-2435-4585-a074-39f4d5d52ec1 node DatanodeRegistration(127.0.0.1:45841, datanodeUuid=13b4a3a9-8383-46bc-9d4a-34a9f4509b76, infoPort=35105, infoSecurePort=0, ipcPort=37475, storageInfo=lv=-57;cid=testClusterID;nsid=817744441;c=1679399728648), blocks: 0, hasStaleStorage: true, processing time: 2 msecs, invalidatedBlocks: 0 2023-03-21 11:55:30,292 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x67811143bcd8a2f9: Processing first storage report for DS-47c72c87-72d5-474f-9416-90b1afef2a44 from datanode 13b4a3a9-8383-46bc-9d4a-34a9f4509b76 2023-03-21 11:55:30,293 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x67811143bcd8a2f9: from storage DS-47c72c87-72d5-474f-9416-90b1afef2a44 node DatanodeRegistration(127.0.0.1:45841, datanodeUuid=13b4a3a9-8383-46bc-9d4a-34a9f4509b76, infoPort=35105, infoSecurePort=0, ipcPort=37475, storageInfo=lv=-57;cid=testClusterID;nsid=817744441;c=1679399728648), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-03-21 11:55:30,320 DEBUG [Listener at localhost.localdomain/37475] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/bb9d57e0-7c01-5166-0245-f9bc94a789fd 2023-03-21 11:55:30,367 INFO [Listener at localhost.localdomain/37475] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/bb9d57e0-7c01-5166-0245-f9bc94a789fd/cluster_9b0830d2-29fd-b47c-c103-f4aba275458e/zookeeper_0, clientPort=61761, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/bb9d57e0-7c01-5166-0245-f9bc94a789fd/cluster_9b0830d2-29fd-b47c-c103-f4aba275458e/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/bb9d57e0-7c01-5166-0245-f9bc94a789fd/cluster_9b0830d2-29fd-b47c-c103-f4aba275458e/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-03-21 11:55:30,379 INFO [Listener at localhost.localdomain/37475] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=61761 2023-03-21 11:55:30,385 INFO [Listener at localhost.localdomain/37475] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-21 11:55:30,387 INFO [Listener at localhost.localdomain/37475] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-21 11:55:30,978 INFO [Listener at localhost.localdomain/37475] util.FSUtils(471): Created version file at hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c with version=8 2023-03-21 11:55:30,978 INFO [Listener at localhost.localdomain/37475] hbase.HBaseTestingUtility(1406): Setting hbase.fs.tmp.dir to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/hbase-staging 2023-03-21 11:55:31,231 INFO [Listener at localhost.localdomain/37475] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-03-21 11:55:31,586 INFO [Listener at localhost.localdomain/37475] client.ConnectionUtils(127): master/jenkins-hbase20:0 server-side Connection retries=45 2023-03-21 11:55:31,610 INFO [Listener at localhost.localdomain/37475] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-03-21 11:55:31,611 INFO [Listener at localhost.localdomain/37475] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-03-21 11:55:31,611 INFO [Listener at localhost.localdomain/37475] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-03-21 11:55:31,611 INFO [Listener at localhost.localdomain/37475] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-03-21 11:55:31,611 INFO [Listener at localhost.localdomain/37475] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-03-21 11:55:31,740 INFO [Listener at localhost.localdomain/37475] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-03-21 11:55:31,802 DEBUG [Listener at localhost.localdomain/37475] util.ClassSize(228): Using Unsafe to estimate memory layout 2023-03-21 11:55:31,878 INFO [Listener at localhost.localdomain/37475] ipc.NettyRpcServer(120): Bind to /148.251.75.209:34627 2023-03-21 11:55:31,887 INFO [Listener at localhost.localdomain/37475] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-21 11:55:31,889 INFO [Listener at localhost.localdomain/37475] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-21 11:55:31,906 INFO [Listener at localhost.localdomain/37475] zookeeper.RecoverableZooKeeper(93): Process identifier=master:34627 connecting to ZooKeeper ensemble=127.0.0.1:61761 2023-03-21 11:55:31,935 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:346270x0, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-03-21 11:55:31,937 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): master:34627-0x10033bc54df0000 connected 2023-03-21 11:55:31,958 DEBUG [Listener at localhost.localdomain/37475] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-03-21 11:55:31,959 DEBUG [Listener at localhost.localdomain/37475] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-03-21 11:55:31,961 DEBUG [Listener at localhost.localdomain/37475] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-03-21 11:55:31,967 DEBUG [Listener at localhost.localdomain/37475] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=34627 2023-03-21 11:55:31,968 DEBUG [Listener at localhost.localdomain/37475] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=34627 2023-03-21 11:55:31,968 DEBUG [Listener at localhost.localdomain/37475] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=34627 2023-03-21 11:55:31,968 DEBUG [Listener at localhost.localdomain/37475] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=34627 2023-03-21 11:55:31,969 DEBUG [Listener at localhost.localdomain/37475] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=34627 2023-03-21 11:55:31,974 INFO [Listener at localhost.localdomain/37475] master.HMaster(444): hbase.rootdir=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c, hbase.cluster.distributed=false 2023-03-21 11:55:32,031 INFO [Listener at localhost.localdomain/37475] client.ConnectionUtils(127): regionserver/jenkins-hbase20:0 server-side Connection retries=45 2023-03-21 11:55:32,031 INFO [Listener at localhost.localdomain/37475] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-03-21 11:55:32,032 INFO [Listener at localhost.localdomain/37475] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-03-21 11:55:32,032 INFO [Listener at localhost.localdomain/37475] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-03-21 11:55:32,032 INFO [Listener at localhost.localdomain/37475] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-03-21 11:55:32,032 INFO [Listener at localhost.localdomain/37475] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-03-21 11:55:32,036 INFO [Listener at localhost.localdomain/37475] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-03-21 11:55:32,039 INFO [Listener at localhost.localdomain/37475] ipc.NettyRpcServer(120): Bind to /148.251.75.209:34369 2023-03-21 11:55:32,041 INFO [Listener at localhost.localdomain/37475] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-03-21 11:55:32,045 DEBUG [Listener at localhost.localdomain/37475] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-03-21 11:55:32,046 INFO [Listener at localhost.localdomain/37475] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-21 11:55:32,049 INFO [Listener at localhost.localdomain/37475] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-21 11:55:32,051 INFO [Listener at localhost.localdomain/37475] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:34369 connecting to ZooKeeper ensemble=127.0.0.1:61761 2023-03-21 11:55:32,054 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:343690x0, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-03-21 11:55:32,055 DEBUG [Listener at localhost.localdomain/37475] zookeeper.ZKUtil(164): regionserver:343690x0, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-03-21 11:55:32,055 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:34369-0x10033bc54df0001 connected 2023-03-21 11:55:32,056 DEBUG [Listener at localhost.localdomain/37475] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-03-21 11:55:32,057 DEBUG [Listener at localhost.localdomain/37475] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-03-21 11:55:32,058 DEBUG [Listener at localhost.localdomain/37475] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=34369 2023-03-21 11:55:32,058 DEBUG [Listener at localhost.localdomain/37475] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=34369 2023-03-21 11:55:32,061 DEBUG [Listener at localhost.localdomain/37475] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=34369 2023-03-21 11:55:32,062 DEBUG [Listener at localhost.localdomain/37475] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=34369 2023-03-21 11:55:32,062 DEBUG [Listener at localhost.localdomain/37475] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=34369 2023-03-21 11:55:32,064 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase20.apache.org,34627,1679399731089 2023-03-21 11:55:32,071 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-03-21 11:55:32,072 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase20.apache.org,34627,1679399731089 2023-03-21 11:55:32,089 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-03-21 11:55:32,089 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-03-21 11:55:32,090 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-21 11:55:32,090 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-03-21 11:55:32,091 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase20.apache.org,34627,1679399731089 from backup master directory 2023-03-21 11:55:32,091 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-03-21 11:55:32,096 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase20.apache.org,34627,1679399731089 2023-03-21 11:55:32,096 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-03-21 11:55:32,097 WARN [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-03-21 11:55:32,097 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase20.apache.org,34627,1679399731089 2023-03-21 11:55:32,099 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating data MemStoreChunkPool with chunk size 2 MB, max count 352, initial count 0 2023-03-21 11:55:32,100 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 391, initial count 0 2023-03-21 11:55:32,175 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] util.FSUtils(620): Created cluster ID file at hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/hbase.id with ID: 0c1dd8ed-0cf2-4021-ba9f-d2be0fec7d10 2023-03-21 11:55:32,216 INFO [master/jenkins-hbase20:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-21 11:55:32,229 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-21 11:55:32,269 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x51e2041c to 127.0.0.1:61761 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-21 11:55:32,296 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@9d713c0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-21 11:55:32,316 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-21 11:55:32,317 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-03-21 11:55:32,332 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(264): ClientProtocol::create wrong number of arguments, should be hadoop 3.2 or below 2023-03-21 11:55:32,332 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(270): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2023-03-21 11:55:32,334 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(279): can not find SHOULD_REPLICATE flag, should be hadoop 2.x java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.CreateFlag.SHOULD_REPLICATE at java.lang.Enum.valueOf(Enum.java:238) at org.apache.hadoop.fs.CreateFlag.valueOf(CreateFlag.java:63) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.loadShouldReplicateFlag(FanOutOneBlockAsyncDFSOutputHelper.java:277) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:304) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:139) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-03-21 11:55:32,337 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(243): No decryptEncryptedDataEncryptionKey method in DFSClient, should be hadoop version with HDFS-12396 java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo) at java.lang.Class.getDeclaredMethod(Class.java:2130) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelperWithoutHDFS12396(FanOutOneBlockAsyncDFSOutputSaslHelper.java:182) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:241) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:252) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:140) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-03-21 11:55:32,338 INFO [master/jenkins-hbase20:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-03-21 11:55:32,364 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(7693): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/MasterData/data/master/store-tmp 2023-03-21 11:55:32,790 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-21 11:55:32,790 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-03-21 11:55:32,790 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-21 11:55:32,790 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-21 11:55:32,790 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-03-21 11:55:32,790 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-21 11:55:32,790 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-21 11:55:32,790 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-03-21 11:55:32,792 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/MasterData/WALs/jenkins-hbase20.apache.org,34627,1679399731089 2023-03-21 11:55:32,812 INFO [master/jenkins-hbase20:0:becomeActiveMaster] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase20.apache.org%2C34627%2C1679399731089, suffix=, logDir=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/MasterData/WALs/jenkins-hbase20.apache.org,34627,1679399731089, archiveDir=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/MasterData/oldWALs, maxLogs=10 2023-03-21 11:55:32,857 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:45841,DS-6ec069ee-2435-4585-a074-39f4d5d52ec1,DISK] 2023-03-21 11:55:32,864 DEBUG [RS-EventLoopGroup-3-2] asyncfs.ProtobufDecoder(123): Hadoop 3.2 and below use unshaded protobuf. java.lang.ClassNotFoundException: org.apache.hadoop.thirdparty.protobuf.MessageLite at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.ProtobufDecoder.(ProtobufDecoder.java:118) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.processWriteBlockResponse(FanOutOneBlockAsyncDFSOutputHelper.java:340) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$100(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$4.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:424) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:185) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:418) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:476) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:471) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:653) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:691) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-21 11:55:32,914 INFO [master/jenkins-hbase20:0:becomeActiveMaster] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/MasterData/WALs/jenkins-hbase20.apache.org,34627,1679399731089/jenkins-hbase20.apache.org%2C34627%2C1679399731089.1679399732821 2023-03-21 11:55:32,915 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:45841,DS-6ec069ee-2435-4585-a074-39f4d5d52ec1,DISK]] 2023-03-21 11:55:32,915 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-03-21 11:55:32,915 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-21 11:55:32,918 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-03-21 11:55:32,919 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-03-21 11:55:32,969 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-03-21 11:55:32,976 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-03-21 11:55:32,997 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-03-21 11:55:33,011 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-21 11:55:33,018 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-03-21 11:55:33,019 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-03-21 11:55:33,035 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-03-21 11:55:33,039 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-21 11:55:33,040 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72837124, jitterRate=0.08535772562026978}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-03-21 11:55:33,040 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-03-21 11:55:33,041 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-03-21 11:55:33,060 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-03-21 11:55:33,060 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-03-21 11:55:33,062 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-03-21 11:55:33,064 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 1 msec 2023-03-21 11:55:33,089 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 25 msec 2023-03-21 11:55:33,089 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(95): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-03-21 11:55:33,113 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-03-21 11:55:33,117 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-03-21 11:55:33,139 INFO [master/jenkins-hbase20:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-03-21 11:55:33,142 INFO [master/jenkins-hbase20:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-03-21 11:55:33,144 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-03-21 11:55:33,148 INFO [master/jenkins-hbase20:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-03-21 11:55:33,151 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-03-21 11:55:33,153 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-21 11:55:33,155 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-03-21 11:55:33,155 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-03-21 11:55:33,165 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-03-21 11:55:33,168 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-03-21 11:55:33,168 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-03-21 11:55:33,169 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-21 11:55:33,169 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase20.apache.org,34627,1679399731089, sessionid=0x10033bc54df0000, setting cluster-up flag (Was=false) 2023-03-21 11:55:33,182 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-21 11:55:33,186 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-03-21 11:55:33,187 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase20.apache.org,34627,1679399731089 2023-03-21 11:55:33,191 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-21 11:55:33,194 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-03-21 11:55:33,196 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase20.apache.org,34627,1679399731089 2023-03-21 11:55:33,199 WARN [master/jenkins-hbase20:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/.hbase-snapshot/.tmp 2023-03-21 11:55:33,266 INFO [RS:0;jenkins-hbase20:34369] regionserver.HRegionServer(952): ClusterId : 0c1dd8ed-0cf2-4021-ba9f-d2be0fec7d10 2023-03-21 11:55:33,270 DEBUG [RS:0;jenkins-hbase20:34369] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-03-21 11:55:33,275 DEBUG [RS:0;jenkins-hbase20:34369] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-03-21 11:55:33,275 DEBUG [RS:0;jenkins-hbase20:34369] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-03-21 11:55:33,278 DEBUG [RS:0;jenkins-hbase20:34369] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-03-21 11:55:33,280 DEBUG [RS:0;jenkins-hbase20:34369] zookeeper.ReadOnlyZKClient(139): Connect 0x6f18cc63 to 127.0.0.1:61761 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-21 11:55:33,290 DEBUG [RS:0;jenkins-hbase20:34369] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@61df2ff0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-21 11:55:33,291 DEBUG [RS:0;jenkins-hbase20:34369] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@65e0a713, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase20.apache.org/148.251.75.209:0 2023-03-21 11:55:33,296 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-03-21 11:55:33,306 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase20:0, corePoolSize=5, maxPoolSize=5 2023-03-21 11:55:33,306 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase20:0, corePoolSize=5, maxPoolSize=5 2023-03-21 11:55:33,307 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=5, maxPoolSize=5 2023-03-21 11:55:33,307 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=5, maxPoolSize=5 2023-03-21 11:55:33,307 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase20:0, corePoolSize=10, maxPoolSize=10 2023-03-21 11:55:33,307 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-03-21 11:55:33,307 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=2, maxPoolSize=2 2023-03-21 11:55:33,307 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-03-21 11:55:33,309 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1679399763309 2023-03-21 11:55:33,312 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-03-21 11:55:33,314 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-03-21 11:55:33,315 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-03-21 11:55:33,321 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-03-21 11:55:33,322 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-03-21 11:55:33,323 DEBUG [RS:0;jenkins-hbase20:34369] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase20:34369 2023-03-21 11:55:33,328 INFO [RS:0;jenkins-hbase20:34369] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-03-21 11:55:33,329 INFO [RS:0;jenkins-hbase20:34369] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-03-21 11:55:33,329 DEBUG [RS:0;jenkins-hbase20:34369] regionserver.HRegionServer(1023): About to register with Master. 2023-03-21 11:55:33,332 INFO [RS:0;jenkins-hbase20:34369] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase20.apache.org,34627,1679399731089 with isa=jenkins-hbase20.apache.org/148.251.75.209:34369, startcode=1679399732030 2023-03-21 11:55:33,332 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-03-21 11:55:33,332 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-03-21 11:55:33,333 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-03-21 11:55:33,333 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-03-21 11:55:33,333 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-03-21 11:55:33,336 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-03-21 11:55:33,337 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-03-21 11:55:33,337 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-03-21 11:55:33,340 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-03-21 11:55:33,340 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-03-21 11:55:33,346 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.large.0-1679399733342,5,FailOnTimeoutGroup] 2023-03-21 11:55:33,347 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.small.0-1679399733347,5,FailOnTimeoutGroup] 2023-03-21 11:55:33,347 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-03-21 11:55:33,347 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-03-21 11:55:33,349 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-03-21 11:55:33,350 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-03-21 11:55:33,357 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-03-21 11:55:33,357 DEBUG [RS:0;jenkins-hbase20:34369] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-03-21 11:55:33,358 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-03-21 11:55:33,358 INFO [PEWorker-1] regionserver.HRegion(7675): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c 2023-03-21 11:55:33,403 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:34405, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=RegionServerStatusService 2023-03-21 11:55:33,414 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.ServerManager(394): Registering regionserver=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:33,428 DEBUG [RS:0;jenkins-hbase20:34369] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c 2023-03-21 11:55:33,428 DEBUG [RS:0;jenkins-hbase20:34369] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost.localdomain:35467 2023-03-21 11:55:33,428 DEBUG [RS:0;jenkins-hbase20:34369] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-03-21 11:55:33,433 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-03-21 11:55:33,433 DEBUG [RS:0;jenkins-hbase20:34369] zookeeper.ZKUtil(162): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:33,433 WARN [RS:0;jenkins-hbase20:34369] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-03-21 11:55:33,434 INFO [RS:0;jenkins-hbase20:34369] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-03-21 11:55:33,435 DEBUG [RS:0;jenkins-hbase20:34369] regionserver.HRegionServer(1947): logDir=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/WALs/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:33,436 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase20.apache.org,34369,1679399732030] 2023-03-21 11:55:33,444 DEBUG [RS:0;jenkins-hbase20:34369] zookeeper.ZKUtil(162): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:33,452 DEBUG [RS:0;jenkins-hbase20:34369] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-03-21 11:55:33,460 INFO [RS:0;jenkins-hbase20:34369] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-03-21 11:55:33,478 INFO [RS:0;jenkins-hbase20:34369] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-03-21 11:55:33,481 INFO [RS:0;jenkins-hbase20:34369] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-03-21 11:55:33,481 INFO [RS:0;jenkins-hbase20:34369] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-03-21 11:55:33,482 INFO [RS:0;jenkins-hbase20:34369] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-03-21 11:55:33,489 INFO [RS:0;jenkins-hbase20:34369] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-03-21 11:55:33,489 DEBUG [RS:0;jenkins-hbase20:34369] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-03-21 11:55:33,489 DEBUG [RS:0;jenkins-hbase20:34369] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-03-21 11:55:33,490 DEBUG [RS:0;jenkins-hbase20:34369] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-03-21 11:55:33,490 DEBUG [RS:0;jenkins-hbase20:34369] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-03-21 11:55:33,490 DEBUG [RS:0;jenkins-hbase20:34369] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-03-21 11:55:33,490 DEBUG [RS:0;jenkins-hbase20:34369] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0, corePoolSize=2, maxPoolSize=2 2023-03-21 11:55:33,490 DEBUG [RS:0;jenkins-hbase20:34369] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-03-21 11:55:33,490 DEBUG [RS:0;jenkins-hbase20:34369] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-03-21 11:55:33,490 DEBUG [RS:0;jenkins-hbase20:34369] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-03-21 11:55:33,490 DEBUG [RS:0;jenkins-hbase20:34369] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-03-21 11:55:33,491 INFO [RS:0;jenkins-hbase20:34369] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-03-21 11:55:33,491 INFO [RS:0;jenkins-hbase20:34369] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-03-21 11:55:33,492 INFO [RS:0;jenkins-hbase20:34369] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-03-21 11:55:33,504 INFO [RS:0;jenkins-hbase20:34369] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-03-21 11:55:33,506 INFO [RS:0;jenkins-hbase20:34369] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,34369,1679399732030-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-03-21 11:55:33,520 INFO [RS:0;jenkins-hbase20:34369] regionserver.Replication(203): jenkins-hbase20.apache.org,34369,1679399732030 started 2023-03-21 11:55:33,520 INFO [RS:0;jenkins-hbase20:34369] regionserver.HRegionServer(1638): Serving as jenkins-hbase20.apache.org,34369,1679399732030, RpcServer on jenkins-hbase20.apache.org/148.251.75.209:34369, sessionid=0x10033bc54df0001 2023-03-21 11:55:33,520 DEBUG [RS:0;jenkins-hbase20:34369] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-03-21 11:55:33,520 DEBUG [RS:0;jenkins-hbase20:34369] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:33,520 DEBUG [RS:0;jenkins-hbase20:34369] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase20.apache.org,34369,1679399732030' 2023-03-21 11:55:33,520 DEBUG [RS:0;jenkins-hbase20:34369] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:33,521 DEBUG [RS:0;jenkins-hbase20:34369] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:33,522 DEBUG [RS:0;jenkins-hbase20:34369] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-03-21 11:55:33,522 DEBUG [RS:0;jenkins-hbase20:34369] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-03-21 11:55:33,522 DEBUG [RS:0;jenkins-hbase20:34369] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:33,522 DEBUG [RS:0;jenkins-hbase20:34369] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase20.apache.org,34369,1679399732030' 2023-03-21 11:55:33,522 DEBUG [RS:0;jenkins-hbase20:34369] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-03-21 11:55:33,522 DEBUG [RS:0;jenkins-hbase20:34369] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-03-21 11:55:33,523 DEBUG [RS:0;jenkins-hbase20:34369] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-03-21 11:55:33,523 INFO [RS:0;jenkins-hbase20:34369] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-03-21 11:55:33,523 INFO [RS:0;jenkins-hbase20:34369] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-03-21 11:55:33,639 INFO [RS:0;jenkins-hbase20:34369] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase20.apache.org%2C34369%2C1679399732030, suffix=, logDir=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/WALs/jenkins-hbase20.apache.org,34369,1679399732030, archiveDir=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/oldWALs, maxLogs=32 2023-03-21 11:55:33,655 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:45841,DS-6ec069ee-2435-4585-a074-39f4d5d52ec1,DISK] 2023-03-21 11:55:33,658 INFO [RS:0;jenkins-hbase20:34369] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/WALs/jenkins-hbase20.apache.org,34369,1679399732030/jenkins-hbase20.apache.org%2C34369%2C1679399732030.1679399733641 2023-03-21 11:55:33,658 DEBUG [RS:0;jenkins-hbase20:34369] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:45841,DS-6ec069ee-2435-4585-a074-39f4d5d52ec1,DISK]] 2023-03-21 11:55:33,776 DEBUG [PEWorker-1] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-21 11:55:33,781 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-03-21 11:55:33,786 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/hbase/meta/1588230740/info 2023-03-21 11:55:33,786 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-03-21 11:55:33,788 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-21 11:55:33,788 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-03-21 11:55:33,790 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/hbase/meta/1588230740/rep_barrier 2023-03-21 11:55:33,791 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-03-21 11:55:33,792 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-21 11:55:33,792 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-03-21 11:55:33,794 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/hbase/meta/1588230740/table 2023-03-21 11:55:33,794 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-03-21 11:55:33,795 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-21 11:55:33,796 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/hbase/meta/1588230740 2023-03-21 11:55:33,797 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/hbase/meta/1588230740 2023-03-21 11:55:33,800 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-21 11:55:33,802 DEBUG [PEWorker-1] regionserver.HRegion(1055): writing seq id for 1588230740 2023-03-21 11:55:33,805 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-21 11:55:33,806 INFO [PEWorker-1] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74910725, jitterRate=0.11625678837299347}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-21 11:55:33,806 DEBUG [PEWorker-1] regionserver.HRegion(965): Region open journal for 1588230740: 2023-03-21 11:55:33,806 DEBUG [PEWorker-1] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-03-21 11:55:33,806 INFO [PEWorker-1] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-03-21 11:55:33,806 DEBUG [PEWorker-1] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-03-21 11:55:33,806 DEBUG [PEWorker-1] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-03-21 11:55:33,807 DEBUG [PEWorker-1] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-03-21 11:55:33,808 INFO [PEWorker-1] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-03-21 11:55:33,808 DEBUG [PEWorker-1] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-03-21 11:55:33,814 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-03-21 11:55:33,814 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-03-21 11:55:33,821 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-03-21 11:55:33,832 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-03-21 11:55:33,834 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-03-21 11:55:33,986 DEBUG [jenkins-hbase20:34627] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-03-21 11:55:33,990 INFO [PEWorker-3] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase20.apache.org,34369,1679399732030, state=OPENING 2023-03-21 11:55:34,000 DEBUG [PEWorker-3] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-03-21 11:55:34,001 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-21 11:55:34,002 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-03-21 11:55:34,004 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase20.apache.org,34369,1679399732030}] 2023-03-21 11:55:34,190 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:34,192 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-03-21 11:55:34,195 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:54780, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-03-21 11:55:34,207 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open hbase:meta,,1.1588230740 2023-03-21 11:55:34,208 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-03-21 11:55:34,212 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase20.apache.org%2C34369%2C1679399732030.meta, suffix=.meta, logDir=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/WALs/jenkins-hbase20.apache.org,34369,1679399732030, archiveDir=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/oldWALs, maxLogs=32 2023-03-21 11:55:34,229 DEBUG [RS-EventLoopGroup-3-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:45841,DS-6ec069ee-2435-4585-a074-39f4d5d52ec1,DISK] 2023-03-21 11:55:34,234 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/WALs/jenkins-hbase20.apache.org,34369,1679399732030/jenkins-hbase20.apache.org%2C34369%2C1679399732030.meta.1679399734213.meta 2023-03-21 11:55:34,234 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:45841,DS-6ec069ee-2435-4585-a074-39f4d5d52ec1,DISK]] 2023-03-21 11:55:34,235 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-03-21 11:55:34,236 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-03-21 11:55:34,254 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-03-21 11:55:34,258 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-03-21 11:55:34,263 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-03-21 11:55:34,263 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-21 11:55:34,263 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-03-21 11:55:34,263 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-03-21 11:55:34,266 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-03-21 11:55:34,268 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/hbase/meta/1588230740/info 2023-03-21 11:55:34,268 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/hbase/meta/1588230740/info 2023-03-21 11:55:34,268 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-03-21 11:55:34,269 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-21 11:55:34,269 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-03-21 11:55:34,271 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/hbase/meta/1588230740/rep_barrier 2023-03-21 11:55:34,271 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/hbase/meta/1588230740/rep_barrier 2023-03-21 11:55:34,271 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-03-21 11:55:34,272 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-21 11:55:34,272 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-03-21 11:55:34,274 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/hbase/meta/1588230740/table 2023-03-21 11:55:34,274 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/hbase/meta/1588230740/table 2023-03-21 11:55:34,274 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-03-21 11:55:34,275 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-21 11:55:34,277 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/hbase/meta/1588230740 2023-03-21 11:55:34,279 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/hbase/meta/1588230740 2023-03-21 11:55:34,282 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-21 11:55:34,284 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-03-21 11:55:34,286 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65108790, jitterRate=-0.029803425073623657}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-21 11:55:34,286 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-03-21 11:55:34,294 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1679399734184 2023-03-21 11:55:34,309 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:meta,,1.1588230740 2023-03-21 11:55:34,310 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened hbase:meta,,1.1588230740 2023-03-21 11:55:34,310 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase20.apache.org,34369,1679399732030, state=OPEN 2023-03-21 11:55:34,312 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-03-21 11:55:34,312 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-03-21 11:55:34,316 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-03-21 11:55:34,317 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase20.apache.org,34369,1679399732030 in 308 msec 2023-03-21 11:55:34,321 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-03-21 11:55:34,322 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 496 msec 2023-03-21 11:55:34,328 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 1.0880 sec 2023-03-21 11:55:34,328 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1679399734328, completionTime=-1 2023-03-21 11:55:34,329 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=1; waited=0ms, expected min=1 server(s), max=1 server(s), master is running 2023-03-21 11:55:34,329 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-03-21 11:55:34,381 DEBUG [hconnection-0x7f4b7c22-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-21 11:55:34,384 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:54794, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-21 11:55:34,399 INFO [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-03-21 11:55:34,399 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1679399794399 2023-03-21 11:55:34,399 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1679399854399 2023-03-21 11:55:34,400 INFO [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 70 msec 2023-03-21 11:55:34,423 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,34627,1679399731089-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-03-21 11:55:34,423 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,34627,1679399731089-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-03-21 11:55:34,423 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,34627,1679399731089-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-03-21 11:55:34,425 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase20:34627, period=300000, unit=MILLISECONDS is enabled. 2023-03-21 11:55:34,425 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-03-21 11:55:34,431 DEBUG [master/jenkins-hbase20:0.Chore.1] janitor.CatalogJanitor(175): 2023-03-21 11:55:34,439 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-03-21 11:55:34,440 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(2148): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-03-21 11:55:34,450 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-03-21 11:55:34,454 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-03-21 11:55:34,457 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-03-21 11:55:34,482 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/.tmp/data/hbase/namespace/2d66eb2b02a1587c2e82d321ca36ece9 2023-03-21 11:55:34,484 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/.tmp/data/hbase/namespace/2d66eb2b02a1587c2e82d321ca36ece9 empty. 2023-03-21 11:55:34,485 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/.tmp/data/hbase/namespace/2d66eb2b02a1587c2e82d321ca36ece9 2023-03-21 11:55:34,485 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-03-21 11:55:34,913 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-03-21 11:55:34,917 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7675): creating {ENCODED => 2d66eb2b02a1587c2e82d321ca36ece9, NAME => 'hbase:namespace,,1679399734440.2d66eb2b02a1587c2e82d321ca36ece9.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/.tmp 2023-03-21 11:55:34,936 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1679399734440.2d66eb2b02a1587c2e82d321ca36ece9.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-21 11:55:34,936 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1604): Closing 2d66eb2b02a1587c2e82d321ca36ece9, disabling compactions & flushes 2023-03-21 11:55:34,936 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1679399734440.2d66eb2b02a1587c2e82d321ca36ece9. 2023-03-21 11:55:34,937 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1679399734440.2d66eb2b02a1587c2e82d321ca36ece9. 2023-03-21 11:55:34,938 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1679399734440.2d66eb2b02a1587c2e82d321ca36ece9. after waiting 0 ms 2023-03-21 11:55:34,938 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1679399734440.2d66eb2b02a1587c2e82d321ca36ece9. 2023-03-21 11:55:34,938 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1838): Closed hbase:namespace,,1679399734440.2d66eb2b02a1587c2e82d321ca36ece9. 2023-03-21 11:55:34,939 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1558): Region close journal for 2d66eb2b02a1587c2e82d321ca36ece9: 2023-03-21 11:55:34,944 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-03-21 11:55:34,960 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1679399734440.2d66eb2b02a1587c2e82d321ca36ece9.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1679399734948"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679399734948"}]},"ts":"1679399734948"} 2023-03-21 11:55:34,983 INFO [PEWorker-4] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-03-21 11:55:34,985 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-03-21 11:55:34,990 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679399734985"}]},"ts":"1679399734985"} 2023-03-21 11:55:34,995 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-03-21 11:55:35,003 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=2d66eb2b02a1587c2e82d321ca36ece9, ASSIGN}] 2023-03-21 11:55:35,006 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=2d66eb2b02a1587c2e82d321ca36ece9, ASSIGN 2023-03-21 11:55:35,008 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=2d66eb2b02a1587c2e82d321ca36ece9, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,34369,1679399732030; forceNewPlan=false, retain=false 2023-03-21 11:55:35,160 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=2d66eb2b02a1587c2e82d321ca36ece9, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:35,160 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1679399734440.2d66eb2b02a1587c2e82d321ca36ece9.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1679399735159"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1679399735159"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679399735159"}]},"ts":"1679399735159"} 2023-03-21 11:55:35,167 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure 2d66eb2b02a1587c2e82d321ca36ece9, server=jenkins-hbase20.apache.org,34369,1679399732030}] 2023-03-21 11:55:35,336 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open hbase:namespace,,1679399734440.2d66eb2b02a1587c2e82d321ca36ece9. 2023-03-21 11:55:35,337 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 2d66eb2b02a1587c2e82d321ca36ece9, NAME => 'hbase:namespace,,1679399734440.2d66eb2b02a1587c2e82d321ca36ece9.', STARTKEY => '', ENDKEY => ''} 2023-03-21 11:55:35,338 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace 2d66eb2b02a1587c2e82d321ca36ece9 2023-03-21 11:55:35,339 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1679399734440.2d66eb2b02a1587c2e82d321ca36ece9.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-21 11:55:35,339 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 2d66eb2b02a1587c2e82d321ca36ece9 2023-03-21 11:55:35,339 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 2d66eb2b02a1587c2e82d321ca36ece9 2023-03-21 11:55:35,342 INFO [StoreOpener-2d66eb2b02a1587c2e82d321ca36ece9-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 2d66eb2b02a1587c2e82d321ca36ece9 2023-03-21 11:55:35,344 DEBUG [StoreOpener-2d66eb2b02a1587c2e82d321ca36ece9-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/hbase/namespace/2d66eb2b02a1587c2e82d321ca36ece9/info 2023-03-21 11:55:35,344 DEBUG [StoreOpener-2d66eb2b02a1587c2e82d321ca36ece9-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/hbase/namespace/2d66eb2b02a1587c2e82d321ca36ece9/info 2023-03-21 11:55:35,345 INFO [StoreOpener-2d66eb2b02a1587c2e82d321ca36ece9-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 2d66eb2b02a1587c2e82d321ca36ece9 columnFamilyName info 2023-03-21 11:55:35,345 INFO [StoreOpener-2d66eb2b02a1587c2e82d321ca36ece9-1] regionserver.HStore(310): Store=2d66eb2b02a1587c2e82d321ca36ece9/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-21 11:55:35,347 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/hbase/namespace/2d66eb2b02a1587c2e82d321ca36ece9 2023-03-21 11:55:35,348 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/hbase/namespace/2d66eb2b02a1587c2e82d321ca36ece9 2023-03-21 11:55:35,352 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 2d66eb2b02a1587c2e82d321ca36ece9 2023-03-21 11:55:35,357 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/hbase/namespace/2d66eb2b02a1587c2e82d321ca36ece9/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-21 11:55:35,358 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 2d66eb2b02a1587c2e82d321ca36ece9; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65947540, jitterRate=-0.017305076122283936}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-03-21 11:55:35,358 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 2d66eb2b02a1587c2e82d321ca36ece9: 2023-03-21 11:55:35,359 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:namespace,,1679399734440.2d66eb2b02a1587c2e82d321ca36ece9., pid=6, masterSystemTime=1679399735323 2023-03-21 11:55:35,363 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:namespace,,1679399734440.2d66eb2b02a1587c2e82d321ca36ece9. 2023-03-21 11:55:35,363 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened hbase:namespace,,1679399734440.2d66eb2b02a1587c2e82d321ca36ece9. 2023-03-21 11:55:35,365 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=2d66eb2b02a1587c2e82d321ca36ece9, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:35,366 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1679399734440.2d66eb2b02a1587c2e82d321ca36ece9.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1679399735364"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1679399735364"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1679399735364"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1679399735364"}]},"ts":"1679399735364"} 2023-03-21 11:55:35,374 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-03-21 11:55:35,374 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure 2d66eb2b02a1587c2e82d321ca36ece9, server=jenkins-hbase20.apache.org,34369,1679399732030 in 203 msec 2023-03-21 11:55:35,378 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-03-21 11:55:35,379 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=2d66eb2b02a1587c2e82d321ca36ece9, ASSIGN in 371 msec 2023-03-21 11:55:35,380 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-03-21 11:55:35,380 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679399735380"}]},"ts":"1679399735380"} 2023-03-21 11:55:35,384 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-03-21 11:55:35,387 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-03-21 11:55:35,389 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 946 msec 2023-03-21 11:55:35,455 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-03-21 11:55:35,457 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-03-21 11:55:35,457 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-21 11:55:35,499 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-03-21 11:55:35,515 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-03-21 11:55:35,521 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 32 msec 2023-03-21 11:55:35,533 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-03-21 11:55:35,544 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-03-21 11:55:35,548 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 14 msec 2023-03-21 11:55:35,561 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-03-21 11:55:35,563 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-03-21 11:55:35,563 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 3.466sec 2023-03-21 11:55:35,565 INFO [master/jenkins-hbase20:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-03-21 11:55:35,567 INFO [master/jenkins-hbase20:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-03-21 11:55:35,568 INFO [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-03-21 11:55:35,569 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,34627,1679399731089-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-03-21 11:55:35,571 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,34627,1679399731089-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-03-21 11:55:35,575 DEBUG [Listener at localhost.localdomain/37475] zookeeper.ReadOnlyZKClient(139): Connect 0x3c41175f to 127.0.0.1:61761 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-21 11:55:35,578 DEBUG [Listener at localhost.localdomain/37475] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6ff31823, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-21 11:55:35,582 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-03-21 11:55:35,590 DEBUG [hconnection-0x56fd3a3e-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-21 11:55:35,600 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:54806, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-21 11:55:35,609 INFO [Listener at localhost.localdomain/37475] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase20.apache.org,34627,1679399731089 2023-03-21 11:55:35,627 INFO [Listener at localhost.localdomain/37475] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMixedAtomicity Thread=205, OpenFileDescriptor=470, MaxFileDescriptor=60000, SystemLoadAverage=231, ProcessCount=167, AvailableMemoryMB=9439 2023-03-21 11:55:35,642 DEBUG [Listener at localhost.localdomain/37475] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-03-21 11:55:35,645 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:43544, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-03-21 11:55:35,653 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-21 11:55:35,656 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.HMaster$4(2112): Client=jenkins//148.251.75.209 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-21 11:55:35,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-03-21 11:55:35,660 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-03-21 11:55:35,663 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-03-21 11:55:35,666 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(697): Client=jenkins//148.251.75.209 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 9 2023-03-21 11:55:35,667 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/.tmp/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49 2023-03-21 11:55:35,669 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/.tmp/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49 empty. 2023-03-21 11:55:35,669 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/.tmp/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49 2023-03-21 11:55:35,670 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-21 11:55:35,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-03-21 11:55:35,686 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-21 11:55:35,687 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 1baade11f622687e2c1237b8441afd49, NAME => 'TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/.tmp 2023-03-21 11:55:35,699 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-21 11:55:35,699 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 1baade11f622687e2c1237b8441afd49, disabling compactions & flushes 2023-03-21 11:55:35,699 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:35,699 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:35,699 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. after waiting 0 ms 2023-03-21 11:55:35,699 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:35,699 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:35,699 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 1baade11f622687e2c1237b8441afd49: 2023-03-21 11:55:35,703 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-03-21 11:55:35,705 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679399735704"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679399735704"}]},"ts":"1679399735704"} 2023-03-21 11:55:35,708 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-03-21 11:55:35,710 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-03-21 11:55:35,710 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679399735710"}]},"ts":"1679399735710"} 2023-03-21 11:55:35,712 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-03-21 11:55:35,715 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=1baade11f622687e2c1237b8441afd49, ASSIGN}] 2023-03-21 11:55:35,717 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=1baade11f622687e2c1237b8441afd49, ASSIGN 2023-03-21 11:55:35,718 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=1baade11f622687e2c1237b8441afd49, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,34369,1679399732030; forceNewPlan=false, retain=false 2023-03-21 11:55:35,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-03-21 11:55:35,870 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=1baade11f622687e2c1237b8441afd49, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:35,871 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679399735870"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1679399735870"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679399735870"}]},"ts":"1679399735870"} 2023-03-21 11:55:35,877 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=11, ppid=10, state=RUNNABLE; OpenRegionProcedure 1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030}] 2023-03-21 11:55:35,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-03-21 11:55:36,042 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:36,042 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1baade11f622687e2c1237b8441afd49, NAME => 'TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49.', STARTKEY => '', ENDKEY => ''} 2023-03-21 11:55:36,043 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 1baade11f622687e2c1237b8441afd49 2023-03-21 11:55:36,043 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-21 11:55:36,043 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 1baade11f622687e2c1237b8441afd49 2023-03-21 11:55:36,043 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 1baade11f622687e2c1237b8441afd49 2023-03-21 11:55:36,045 INFO [StoreOpener-1baade11f622687e2c1237b8441afd49-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 1baade11f622687e2c1237b8441afd49 2023-03-21 11:55:36,048 DEBUG [StoreOpener-1baade11f622687e2c1237b8441afd49-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A 2023-03-21 11:55:36,048 DEBUG [StoreOpener-1baade11f622687e2c1237b8441afd49-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A 2023-03-21 11:55:36,049 INFO [StoreOpener-1baade11f622687e2c1237b8441afd49-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-21 11:55:36,050 INFO [StoreOpener-1baade11f622687e2c1237b8441afd49-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1baade11f622687e2c1237b8441afd49 columnFamilyName A 2023-03-21 11:55:36,051 INFO [StoreOpener-1baade11f622687e2c1237b8441afd49-1] regionserver.HStore(310): Store=1baade11f622687e2c1237b8441afd49/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-21 11:55:36,051 INFO [StoreOpener-1baade11f622687e2c1237b8441afd49-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 1baade11f622687e2c1237b8441afd49 2023-03-21 11:55:36,053 DEBUG [StoreOpener-1baade11f622687e2c1237b8441afd49-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B 2023-03-21 11:55:36,053 DEBUG [StoreOpener-1baade11f622687e2c1237b8441afd49-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B 2023-03-21 11:55:36,053 INFO [StoreOpener-1baade11f622687e2c1237b8441afd49-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-21 11:55:36,054 INFO [StoreOpener-1baade11f622687e2c1237b8441afd49-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1baade11f622687e2c1237b8441afd49 columnFamilyName B 2023-03-21 11:55:36,055 INFO [StoreOpener-1baade11f622687e2c1237b8441afd49-1] regionserver.HStore(310): Store=1baade11f622687e2c1237b8441afd49/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-21 11:55:36,055 INFO [StoreOpener-1baade11f622687e2c1237b8441afd49-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 1baade11f622687e2c1237b8441afd49 2023-03-21 11:55:36,057 DEBUG [StoreOpener-1baade11f622687e2c1237b8441afd49-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C 2023-03-21 11:55:36,057 DEBUG [StoreOpener-1baade11f622687e2c1237b8441afd49-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C 2023-03-21 11:55:36,057 INFO [StoreOpener-1baade11f622687e2c1237b8441afd49-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-21 11:55:36,057 INFO [StoreOpener-1baade11f622687e2c1237b8441afd49-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1baade11f622687e2c1237b8441afd49 columnFamilyName C 2023-03-21 11:55:36,058 INFO [StoreOpener-1baade11f622687e2c1237b8441afd49-1] regionserver.HStore(310): Store=1baade11f622687e2c1237b8441afd49/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-21 11:55:36,059 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:36,061 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49 2023-03-21 11:55:36,061 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49 2023-03-21 11:55:36,065 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-21 11:55:36,067 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 1baade11f622687e2c1237b8441afd49 2023-03-21 11:55:36,070 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-21 11:55:36,071 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 1baade11f622687e2c1237b8441afd49; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67533039, jitterRate=0.00632070004940033}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-21 11:55:36,071 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 1baade11f622687e2c1237b8441afd49: 2023-03-21 11:55:36,073 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49., pid=11, masterSystemTime=1679399736031 2023-03-21 11:55:36,075 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:36,075 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:36,076 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=1baade11f622687e2c1237b8441afd49, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:36,077 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679399736076"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1679399736076"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1679399736076"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1679399736076"}]},"ts":"1679399736076"} 2023-03-21 11:55:36,083 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=11, resume processing ppid=10 2023-03-21 11:55:36,084 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=10, state=SUCCESS; OpenRegionProcedure 1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 in 203 msec 2023-03-21 11:55:36,088 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=10, resume processing ppid=9 2023-03-21 11:55:36,088 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=1baade11f622687e2c1237b8441afd49, ASSIGN in 369 msec 2023-03-21 11:55:36,090 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-03-21 11:55:36,090 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679399736090"}]},"ts":"1679399736090"} 2023-03-21 11:55:36,092 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-03-21 11:55:36,095 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-03-21 11:55:36,097 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 439 msec 2023-03-21 11:55:36,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-03-21 11:55:36,291 INFO [Listener at localhost.localdomain/37475] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 9 completed 2023-03-21 11:55:36,300 DEBUG [Listener at localhost.localdomain/37475] zookeeper.ReadOnlyZKClient(139): Connect 0x4a427c73 to 127.0.0.1:61761 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-21 11:55:36,304 DEBUG [Listener at localhost.localdomain/37475] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@380b74b4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-21 11:55:36,307 DEBUG [Listener at localhost.localdomain/37475] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-21 11:55:36,311 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:54812, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-21 11:55:36,315 DEBUG [Listener at localhost.localdomain/37475] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-03-21 11:55:36,318 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:43554, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-03-21 11:55:36,325 DEBUG [Listener at localhost.localdomain/37475] zookeeper.ReadOnlyZKClient(139): Connect 0x78c6cbc8 to 127.0.0.1:61761 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-21 11:55:36,330 DEBUG [Listener at localhost.localdomain/37475] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5e6ad943, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-21 11:55:36,330 DEBUG [Listener at localhost.localdomain/37475] zookeeper.ReadOnlyZKClient(139): Connect 0x3b15648c to 127.0.0.1:61761 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-21 11:55:36,334 DEBUG [Listener at localhost.localdomain/37475] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7d6b0f87, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-21 11:55:36,335 DEBUG [Listener at localhost.localdomain/37475] zookeeper.ReadOnlyZKClient(139): Connect 0x3339866b to 127.0.0.1:61761 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-21 11:55:36,339 DEBUG [Listener at localhost.localdomain/37475] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5d114eb9, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-21 11:55:36,339 DEBUG [Listener at localhost.localdomain/37475] zookeeper.ReadOnlyZKClient(139): Connect 0x1ee17908 to 127.0.0.1:61761 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-21 11:55:36,343 DEBUG [Listener at localhost.localdomain/37475] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3627406c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-21 11:55:36,343 DEBUG [Listener at localhost.localdomain/37475] zookeeper.ReadOnlyZKClient(139): Connect 0x40f66fa6 to 127.0.0.1:61761 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-21 11:55:36,347 DEBUG [Listener at localhost.localdomain/37475] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4d5a66f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-21 11:55:36,348 DEBUG [Listener at localhost.localdomain/37475] zookeeper.ReadOnlyZKClient(139): Connect 0x31222f6c to 127.0.0.1:61761 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-21 11:55:36,355 DEBUG [Listener at localhost.localdomain/37475] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@795a2478, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-21 11:55:36,356 DEBUG [Listener at localhost.localdomain/37475] zookeeper.ReadOnlyZKClient(139): Connect 0x29fd8399 to 127.0.0.1:61761 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-21 11:55:36,388 DEBUG [Listener at localhost.localdomain/37475] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@9048085, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-21 11:55:36,389 DEBUG [Listener at localhost.localdomain/37475] zookeeper.ReadOnlyZKClient(139): Connect 0x57966649 to 127.0.0.1:61761 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-21 11:55:36,397 DEBUG [Listener at localhost.localdomain/37475] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7abbc496, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-21 11:55:36,397 DEBUG [Listener at localhost.localdomain/37475] zookeeper.ReadOnlyZKClient(139): Connect 0x7746d7a7 to 127.0.0.1:61761 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-21 11:55:36,407 DEBUG [Listener at localhost.localdomain/37475] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@13065328, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-21 11:55:36,415 DEBUG [hconnection-0xe098c00-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-21 11:55:36,418 DEBUG [hconnection-0x76a67d5a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-21 11:55:36,422 DEBUG [hconnection-0x9ae6b3d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-21 11:55:36,422 DEBUG [hconnection-0x7c96301-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-21 11:55:36,422 DEBUG [hconnection-0x1923f300-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-21 11:55:36,422 DEBUG [hconnection-0x6ad481a8-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-21 11:55:36,445 DEBUG [hconnection-0x7c9c57b5-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-21 11:55:36,445 DEBUG [hconnection-0x36681d36-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-21 11:55:36,445 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:54828, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-21 11:55:36,446 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:54836, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-21 11:55:36,448 DEBUG [hconnection-0x34432ac-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-21 11:55:36,449 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:54840, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-21 11:55:36,449 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:54864, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-21 11:55:36,449 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:54850, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-21 11:55:36,458 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:55:36,460 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:54892, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-21 11:55:36,460 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:54894, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-21 11:55:36,463 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:54878, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-21 11:55:36,463 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:54904, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-21 11:55:36,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:55:36,476 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:55:36,476 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:36,486 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:55:36,487 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:55:36,487 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:36,487 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:36,489 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:36,489 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:36,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:36,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:36,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:36,490 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:36,490 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:55:36,490 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:36,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:55:36,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:36,491 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:55:36,494 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:55:36,495 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:55:36,495 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:36,496 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:55:36,497 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:55:36,497 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:36,497 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:55:36,498 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. started... 2023-03-21 11:55:36,499 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1baade11f622687e2c1237b8441afd49 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-03-21 11:55:36,510 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1baade11f622687e2c1237b8441afd49, store=A 2023-03-21 11:55:36,512 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:55:36,512 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1baade11f622687e2c1237b8441afd49, store=B 2023-03-21 11:55:36,512 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:55:36,513 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1baade11f622687e2c1237b8441afd49, store=C 2023-03-21 11:55:36,513 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:55:36,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(9158): Flush requested on 1baade11f622687e2c1237b8441afd49 2023-03-21 11:55:36,527 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. as already flushing 2023-03-21 11:55:36,617 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679399736571, max timestamp=1679399736590Num uniques 859; 2023-03-21 11:55:36,626 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-21 11:55:36,629 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679399736576, max timestamp=1679399736590Num uniques 859; 2023-03-21 11:55:36,630 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-03-21 11:55:36,635 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679399736576, max timestamp=1679399736591Num uniques 859; 2023-03-21 11:55:36,635 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-03-21 11:55:36,689 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:36,689 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1679399736667, max timestamp=1679399736681Num uniques 1118; 2023-03-21 11:55:36,690 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=12268, suffix heap size=144240,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=268 2023-03-21 11:55:36,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54904 deadline: 1679399796675, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:36,691 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679399736669, max timestamp=1679399736681Num uniques 1009; 2023-03-21 11:55:36,691 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=46213, new segment data size=6870, suffix heap size=132904,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=150 2023-03-21 11:55:36,697 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1679399736667, max timestamp=1679399736681Num uniques 1009; 2023-03-21 11:55:36,697 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=46213, new segment data size=12268, suffix heap size=132904,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=268 2023-03-21 11:55:36,771 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:36,772 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679399736725, max timestamp=1679399736744Num uniques 1127; 2023-03-21 11:55:36,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54894 deadline: 1679399796743, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:36,773 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=51602, new segment data size=6870, suffix heap size=145176,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=150 2023-03-21 11:55:36,775 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679399736725, max timestamp=1679399736744Num uniques 1127; 2023-03-21 11:55:36,775 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=6870, suffix heap size=145176,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=150 2023-03-21 11:55:36,778 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679399736725, max timestamp=1679399736744Num uniques 1009; 2023-03-21 11:55:36,778 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=46204, new segment data size=6870, suffix heap size=132904,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=150 2023-03-21 11:55:36,827 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:36,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54904 deadline: 1679399796818, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:36,828 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1679399736795, max timestamp=1679399736822Num uniques 1009; 2023-03-21 11:55:36,828 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=46224, new segment data size=8516, suffix heap size=132904,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=186 2023-03-21 11:55:36,828 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1679399736795, max timestamp=1679399736821Num uniques 1009; 2023-03-21 11:55:36,830 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=8516, suffix heap size=132904,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=186 2023-03-21 11:55:36,841 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679399736795, max timestamp=1679399736816Num uniques 1009; 2023-03-21 11:55:36,841 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=46224, new segment data size=6870, suffix heap size=132904,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=150 2023-03-21 11:55:36,884 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1679399736870, max timestamp=1679399736879Num uniques 1009; 2023-03-21 11:55:36,884 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=8930, suffix heap size=132904,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=195 2023-03-21 11:55:36,884 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1679399736870, max timestamp=1679399736879Num uniques 1045; 2023-03-21 11:55:36,884 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-03-21 11:55:36,885 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1679399736870, max timestamp=1679399736879Num uniques 1045; 2023-03-21 11:55:36,885 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-03-21 11:55:36,924 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:36,924 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1679399736910, max timestamp=1679399736920Num uniques 1054; 2023-03-21 11:55:36,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54864 deadline: 1679399796921, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:36,925 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-03-21 11:55:36,924 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1679399736910, max timestamp=1679399736920Num uniques 1054; 2023-03-21 11:55:36,925 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-03-21 11:55:36,924 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1679399736910, max timestamp=1679399736920Num uniques 1054; 2023-03-21 11:55:36,927 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-03-21 11:55:37,017 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=263, cellSize=12038, totalHeapSize=27592, min timestamp=1679399736993, max timestamp=1679399737014Num uniques 1013; 2023-03-21 11:55:37,017 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=12038, suffix heap size=133320,new segment heap size=27592 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=263 2023-03-21 11:55:37,018 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=263, cellSize=12038, totalHeapSize=27592, min timestamp=1679399736993, max timestamp=1679399737014Num uniques 1013; 2023-03-21 11:55:37,018 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=12038, suffix heap size=133320,new segment heap size=27592 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=263 2023-03-21 11:55:37,019 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=263, cellSize=12038, totalHeapSize=27592, min timestamp=1679399736993, max timestamp=1679399737014Num uniques 1013; 2023-03-21 11:55:37,020 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=12038, suffix heap size=133320,new segment heap size=27592 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=263 2023-03-21 11:55:37,067 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:37,068 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=9 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/A/1ab8e06c6cf6468db0b6a9f3d60d1717 2023-03-21 11:55:37,069 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1679399737023, max timestamp=1679399737064Num uniques 1122; 2023-03-21 11:55:37,069 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=51382, new segment data size=7872, suffix heap size=144656,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1122, new segment cells count=172 2023-03-21 11:55:37,069 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1679399737023, max timestamp=1679399737064Num uniques 1122; 2023-03-21 11:55:37,070 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=51382, new segment data size=7872, suffix heap size=144656,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1122, new segment cells count=172 2023-03-21 11:55:37,067 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:37,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54894 deadline: 1679399797058, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:37,072 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1679399737023, max timestamp=1679399737064Num uniques 1122; 2023-03-21 11:55:37,072 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=51382, new segment data size=7872, suffix heap size=144656,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1122, new segment cells count=172 2023-03-21 11:55:37,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54864 deadline: 1679399797056, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:37,134 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1679399737098, max timestamp=1679399737130Num uniques 1031; 2023-03-21 11:55:37,135 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=8286, suffix heap size=135192,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=181 2023-03-21 11:55:37,136 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1679399737098, max timestamp=1679399737131Num uniques 1031; 2023-03-21 11:55:37,136 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=8286, suffix heap size=135192,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=181 2023-03-21 11:55:37,137 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1679399737098, max timestamp=1679399737130Num uniques 1031; 2023-03-21 11:55:37,138 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=8286, suffix heap size=135192,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=181 2023-03-21 11:55:37,183 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1679399737161, max timestamp=1679399737176Num uniques 1040; 2023-03-21 11:55:37,183 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=10990, suffix heap size=136128,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=240 2023-03-21 11:55:37,186 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1679399737161, max timestamp=1679399737176Num uniques 1040; 2023-03-21 11:55:37,186 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=10990, suffix heap size=136128,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=240 2023-03-21 11:55:37,187 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1679399737161, max timestamp=1679399737181Num uniques 1040; 2023-03-21 11:55:37,187 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=10990, suffix heap size=136128,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=240 2023-03-21 11:55:37,218 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1679399737201, max timestamp=1679399737206Num uniques 1099; 2023-03-21 11:55:37,218 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=50334, new segment data size=11404, suffix heap size=142264,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=249 2023-03-21 11:55:37,220 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1679399737201, max timestamp=1679399737206Num uniques 1099; 2023-03-21 11:55:37,220 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=50334, new segment data size=9114, suffix heap size=142264,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=199 2023-03-21 11:55:37,222 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1679399737201, max timestamp=1679399737206Num uniques 1099; 2023-03-21 11:55:37,222 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=50334, new segment data size=9114, suffix heap size=142264,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=199 2023-03-21 11:55:37,262 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1679399737247, max timestamp=1679399737255Num uniques 1108; 2023-03-21 11:55:37,263 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=50740, new segment data size=9520, suffix heap size=143200,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1108, new segment cells count=208 2023-03-21 11:55:37,263 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1679399737247, max timestamp=1679399737255Num uniques 1058; 2023-03-21 11:55:37,263 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=9520, suffix heap size=138000,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=208 2023-03-21 11:55:37,263 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1679399737247, max timestamp=1679399737255Num uniques 1058; 2023-03-21 11:55:37,263 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=9520, suffix heap size=138000,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=208 2023-03-21 11:55:37,294 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=167, cellSize=7642, totalHeapSize=17608, min timestamp=1679399737287, max timestamp=1679399737291Num uniques 1067; 2023-03-21 11:55:37,294 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=7642, suffix heap size=138936,new segment heap size=17608 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=167 2023-03-21 11:55:37,296 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=167, cellSize=7642, totalHeapSize=17608, min timestamp=1679399737287, max timestamp=1679399737291Num uniques 1067; 2023-03-21 11:55:37,296 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=7642, suffix heap size=138936,new segment heap size=17608 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=167 2023-03-21 11:55:37,297 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=167, cellSize=7642, totalHeapSize=17608, min timestamp=1679399737287, max timestamp=1679399737291Num uniques 1067; 2023-03-21 11:55:37,297 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=7642, suffix heap size=138936,new segment heap size=17608 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=167 2023-03-21 11:55:37,322 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1679399737303, max timestamp=1679399737318Num uniques 1026; 2023-03-21 11:55:37,322 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1679399737303, max timestamp=1679399737318Num uniques 1026; 2023-03-21 11:55:37,322 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=46986, new segment data size=8056, suffix heap size=134672,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1026, new segment cells count=176 2023-03-21 11:55:37,322 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=46986, new segment data size=8056, suffix heap size=134672,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1026, new segment cells count=176 2023-03-21 11:55:37,324 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1679399737303, max timestamp=1679399737318Num uniques 1026; 2023-03-21 11:55:37,324 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=46986, new segment data size=8056, suffix heap size=134672,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1026, new segment cells count=176 2023-03-21 11:55:37,364 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=185, cellSize=8470, totalHeapSize=19480, min timestamp=1679399737350, max timestamp=1679399737356Num uniques 1035; 2023-03-21 11:55:37,364 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1679399737350, max timestamp=1679399737361Num uniques 1035; 2023-03-21 11:55:37,366 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=47396, new segment data size=7050, suffix heap size=135608,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=154 2023-03-21 11:55:37,365 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:37,364 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=47400, new segment data size=8470, suffix heap size=135608,new segment heap size=19480 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=185 2023-03-21 11:55:37,364 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=185, cellSize=8470, totalHeapSize=19480, min timestamp=1679399737350, max timestamp=1679399737356Num uniques 1035; 2023-03-21 11:55:37,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54894 deadline: 1679399797356, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:37,367 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=47400, new segment data size=8470, suffix heap size=135608,new segment heap size=19480 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=185 2023-03-21 11:55:37,400 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=194, cellSize=8884, totalHeapSize=20416, min timestamp=1679399737384, max timestamp=1679399737395Num uniques 1044; 2023-03-21 11:55:37,400 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=47814, new segment data size=8884, suffix heap size=136544,new segment heap size=20416 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1044, new segment cells count=194 2023-03-21 11:55:37,403 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=194, cellSize=8884, totalHeapSize=20416, min timestamp=1679399737384, max timestamp=1679399737395Num uniques 1044; 2023-03-21 11:55:37,403 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=194, cellSize=8884, totalHeapSize=20416, min timestamp=1679399737384, max timestamp=1679399737395Num uniques 1013; 2023-03-21 11:55:37,403 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=46398, new segment data size=8884, suffix heap size=133320,new segment heap size=20416 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=194 2023-03-21 11:55:37,403 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=47814, new segment data size=8884, suffix heap size=136544,new segment heap size=20416 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1044, new segment cells count=194 2023-03-21 11:55:37,451 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=153, cellSize=7005, totalHeapSize=16152, min timestamp=1679399737432, max timestamp=1679399737447Num uniques 1053; 2023-03-21 11:55:37,451 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=48225, new segment data size=7005, suffix heap size=137480,new segment heap size=16152 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1053, new segment cells count=153 2023-03-21 11:55:37,451 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=153, cellSize=7005, totalHeapSize=16152, min timestamp=1679399737432, max timestamp=1679399737446Num uniques 1053; 2023-03-21 11:55:37,451 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=48225, new segment data size=7005, suffix heap size=137480,new segment heap size=16152 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1053, new segment cells count=153 2023-03-21 11:55:37,451 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=153, cellSize=7005, totalHeapSize=16152, min timestamp=1679399737432, max timestamp=1679399737447Num uniques 1053; 2023-03-21 11:55:37,451 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=48225, new segment data size=7005, suffix heap size=137480,new segment heap size=16152 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1053, new segment cells count=153 2023-03-21 11:55:37,484 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=212, cellSize=9702, totalHeapSize=22288, min timestamp=1679399737467, max timestamp=1679399737481Num uniques 1012; 2023-03-21 11:55:37,484 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=46342, new segment data size=9702, suffix heap size=133216,new segment heap size=22288 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1012, new segment cells count=212 2023-03-21 11:55:37,486 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=212, cellSize=9702, totalHeapSize=22288, min timestamp=1679399737467, max timestamp=1679399737481Num uniques 1012; 2023-03-21 11:55:37,486 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=46342, new segment data size=9702, suffix heap size=133216,new segment heap size=22288 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1012, new segment cells count=212 2023-03-21 11:55:37,484 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=212, cellSize=9702, totalHeapSize=22288, min timestamp=1679399737467, max timestamp=1679399737481Num uniques 1012; 2023-03-21 11:55:37,487 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=46342, new segment data size=9702, suffix heap size=133216,new segment heap size=22288 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1012, new segment cells count=212 2023-03-21 11:55:37,517 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=321, cellSize=14696, totalHeapSize=33624, min timestamp=1679399737494, max timestamp=1679399737514Num uniques 1071; 2023-03-21 11:55:37,518 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=321, cellSize=14696, totalHeapSize=33624, min timestamp=1679399737494, max timestamp=1679399737514Num uniques 1071; 2023-03-21 11:55:37,518 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=49046, new segment data size=14696, suffix heap size=139352,new segment heap size=33624 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1071, new segment cells count=321 2023-03-21 11:55:37,518 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=49046, new segment data size=14696, suffix heap size=139352,new segment heap size=33624 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1071, new segment cells count=321 2023-03-21 11:55:37,520 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=171, cellSize=7826, totalHeapSize=18024, min timestamp=1679399737501, max timestamp=1679399737514Num uniques 1071; 2023-03-21 11:55:37,520 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=49046, new segment data size=7826, suffix heap size=139352,new segment heap size=18024 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1071, new segment cells count=171 2023-03-21 11:55:37,562 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:37,563 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=230, cellSize=10530, totalHeapSize=24160, min timestamp=1679399737537, max timestamp=1679399737559Num uniques 1180; 2023-03-21 11:55:37,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54904 deadline: 1679399797552, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:37,563 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=54040, new segment data size=10530, suffix heap size=150688,new segment heap size=24160 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1180, new segment cells count=230 2023-03-21 11:55:37,563 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=230, cellSize=10530, totalHeapSize=24160, min timestamp=1679399737537, max timestamp=1679399737559Num uniques 1180; 2023-03-21 11:55:37,563 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=54040, new segment data size=10530, suffix heap size=150688,new segment heap size=24160 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1180, new segment cells count=230 2023-03-21 11:55:37,565 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=230, cellSize=10530, totalHeapSize=24160, min timestamp=1679399737537, max timestamp=1679399737559Num uniques 1030; 2023-03-21 11:55:37,565 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=47170, new segment data size=10530, suffix heap size=135088,new segment heap size=24160 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1030, new segment cells count=230 2023-03-21 11:55:37,614 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=289, cellSize=13234, totalHeapSize=30296, min timestamp=1679399737587, max timestamp=1679399737611Num uniques 1089; 2023-03-21 11:55:37,614 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=49874, new segment data size=13234, suffix heap size=141224,new segment heap size=30296 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1089, new segment cells count=289 2023-03-21 11:55:37,615 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=189, cellSize=8654, totalHeapSize=19896, min timestamp=1679399737597, max timestamp=1679399737611Num uniques 1089; 2023-03-21 11:55:37,616 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=49874, new segment data size=8654, suffix heap size=141224,new segment heap size=19896 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1089, new segment cells count=189 2023-03-21 11:55:37,618 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679399737597, max timestamp=1679399737611Num uniques 1089; 2023-03-21 11:55:37,618 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=49874, new segment data size=6870, suffix heap size=141224,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1089, new segment cells count=150 2023-03-21 11:55:37,631 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=9 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/B/0120925148ae4c879ca049603ff834e6 2023-03-21 11:55:37,649 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=298, cellSize=13648, totalHeapSize=31232, min timestamp=1679399737633, max timestamp=1679399737646Num uniques 1148; 2023-03-21 11:55:37,649 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=52578, new segment data size=13648, suffix heap size=147360,new segment heap size=31232 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1148, new segment cells count=298 2023-03-21 11:55:37,651 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=198, cellSize=9068, totalHeapSize=20832, min timestamp=1679399737633, max timestamp=1679399737646Num uniques 1048; 2023-03-21 11:55:37,652 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=47998, new segment data size=9068, suffix heap size=136960,new segment heap size=20832 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1048, new segment cells count=198 2023-03-21 11:55:37,652 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=198, cellSize=9068, totalHeapSize=20832, min timestamp=1679399737633, max timestamp=1679399737646Num uniques 1009; 2023-03-21 11:55:37,652 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=9068, suffix heap size=132904,new segment heap size=20832 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=198 2023-03-21 11:55:37,699 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=9 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/C/d238a1367a424afcaa9c605131083402 2023-03-21 11:55:37,709 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/A/1ab8e06c6cf6468db0b6a9f3d60d1717 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/1ab8e06c6cf6468db0b6a9f3d60d1717 2023-03-21 11:55:37,721 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/1ab8e06c6cf6468db0b6a9f3d60d1717, entries=100, sequenceid=9, filesize=9.3 K 2023-03-21 11:55:37,725 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.StoreScanner(961): StoreScanner already closing. There is no need to updateReaders 2023-03-21 11:55:37,729 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/B/0120925148ae4c879ca049603ff834e6 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/0120925148ae4c879ca049603ff834e6 2023-03-21 11:55:37,756 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/0120925148ae4c879ca049603ff834e6, entries=100, sequenceid=9, filesize=9.3 K 2023-03-21 11:55:37,759 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/C/d238a1367a424afcaa9c605131083402 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/d238a1367a424afcaa9c605131083402 2023-03-21 11:55:37,773 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/d238a1367a424afcaa9c605131083402, entries=100, sequenceid=9, filesize=9.3 K 2023-03-21 11:55:37,776 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=131.94 KB/135110 for 1baade11f622687e2c1237b8441afd49 in 1277ms, sequenceid=9, compaction requested=false 2023-03-21 11:55:37,778 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-03-21 11:55:37,782 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1baade11f622687e2c1237b8441afd49: 2023-03-21 11:55:37,782 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:37,782 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:55:37,783 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:55:37,783 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:37,783 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:55:37,783 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:55:37,789 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:37,790 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:37,790 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:55:37,790 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:37,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:37,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:37,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:37,791 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1baade11f622687e2c1237b8441afd49 3/3 column families, dataSize=138.65 KB heapSize=391.48 KB 2023-03-21 11:55:37,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:37,791 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1baade11f622687e2c1237b8441afd49, store=A 2023-03-21 11:55:37,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:37,792 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-21 11:55:37,792 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1baade11f622687e2c1237b8441afd49, store=B 2023-03-21 11:55:37,792 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-21 11:55:37,792 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1baade11f622687e2c1237b8441afd49, store=C 2023-03-21 11:55:37,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:37,792 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-21 11:55:37,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(9158): Flush requested on 1baade11f622687e2c1237b8441afd49 2023-03-21 11:55:37,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:37,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:37,794 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:55:37,795 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@646c13c0[Count = 0] remaining members to acquire global barrier 2023-03-21 11:55:37,795 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:55:37,795 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:37,810 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:37,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:37,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:37,811 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:37,811 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:55:37,811 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:55:37,811 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:55:37,811 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:55:37,820 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:37,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:37,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:37,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:37,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:37,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:37,821 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:55:37,821 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:37,821 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:55:37,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:37,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:37,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:37,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:37,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:37,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:55:37,829 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:55:37,829 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:55:37,829 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:55:37,829 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:55:37,829 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:55:37,829 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:55:37,840 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:37,840 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:37,840 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:37,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:37,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:37,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:37,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:37,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:37,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:37,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:37,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:37,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:37,842 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:37,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:37,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:37,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:37,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:37,845 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:37,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:37,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:37,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:37,848 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:37,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54840 deadline: 1679399797805, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:37,849 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:37,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54904 deadline: 1679399797807, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:37,864 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:37,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54892 deadline: 1679399797843, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:37,864 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:37,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54894 deadline: 1679399797845, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:37,865 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:37,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54864 deadline: 1679399797846, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:37,865 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:37,865 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:37,866 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:37,866 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:37,866 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:37,866 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:37,866 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:37,866 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:55:37,866 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:55:37,866 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:37,868 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:37,866 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:37,868 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:55:37,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:37,868 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:55:37,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:37,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:37,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:37,910 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=406 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/A/779facedcefe4862b469363e0301cc77 2023-03-21 11:55:37,968 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:37,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54840 deadline: 1679399797967, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:37,970 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:55:37,972 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:37,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54892 deadline: 1679399797971, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:37,972 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:37,972 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:37,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54894 deadline: 1679399797971, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:37,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54864 deadline: 1679399797972, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:37,974 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:37,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54904 deadline: 1679399797974, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:37,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:55:37,985 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:55:38,003 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=406 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/B/da21ab7ad4974bdf929c76397cb2653b 2023-03-21 11:55:38,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:55:38,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:55:38,009 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:55:38,009 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:38,010 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:55:38,010 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:55:38,010 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:38,010 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:38,012 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:38,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:38,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:38,013 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:38,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:38,013 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:38,013 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:55:38,013 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:38,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:55:38,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:38,014 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:55:38,026 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:55:38,026 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:55:38,026 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:55:38,026 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:38,027 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:55:38,028 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:55:38,028 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:55:38,030 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:38,030 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. started... 2023-03-21 11:55:38,042 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. as already flushing 2023-03-21 11:55:38,042 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:38,043 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:55:38,043 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:55:38,043 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:38,043 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:55:38,043 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:55:38,051 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:38,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:38,052 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:38,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:38,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:38,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:38,053 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:38,053 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:55:38,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:38,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:38,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:38,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:38,055 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:55:38,056 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1bc9c463[Count = 0] remaining members to acquire global barrier 2023-03-21 11:55:38,056 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:55:38,056 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:38,057 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:38,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:38,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:38,057 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:55:38,057 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:55:38,057 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:55:38,057 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:38,058 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:55:38,061 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:55:38,061 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:38,062 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:55:38,064 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:38,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:38,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:38,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:38,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:38,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:38,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:38,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:38,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:38,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:38,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:38,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:55:38,073 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:55:38,073 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:55:38,073 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:55:38,073 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:55:38,073 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:55:38,073 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:55:38,084 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:38,084 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:38,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:38,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:38,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:38,084 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:38,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:38,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:38,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:38,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:38,091 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:38,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:38,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:38,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:38,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:38,093 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:38,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:38,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:38,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:38,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:38,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:38,114 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:38,114 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:38,114 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:38,114 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:38,114 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:38,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:38,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:38,114 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:38,114 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:38,114 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=406 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/C/dd688aa38f3845188d8c91c8f59da958 2023-03-21 11:55:38,114 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:38,115 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:38,115 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:38,114 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:55:38,115 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:55:38,115 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:55:38,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:38,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:38,116 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:55:38,116 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:55:38,128 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/A/779facedcefe4862b469363e0301cc77 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/779facedcefe4862b469363e0301cc77 2023-03-21 11:55:38,146 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/779facedcefe4862b469363e0301cc77, entries=450, sequenceid=406, filesize=26.2 K 2023-03-21 11:55:38,149 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/B/da21ab7ad4974bdf929c76397cb2653b as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/da21ab7ad4974bdf929c76397cb2653b 2023-03-21 11:55:38,180 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/da21ab7ad4974bdf929c76397cb2653b, entries=150, sequenceid=406, filesize=11.9 K 2023-03-21 11:55:38,183 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:38,183 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/C/dd688aa38f3845188d8c91c8f59da958 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/dd688aa38f3845188d8c91c8f59da958 2023-03-21 11:55:38,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54840 deadline: 1679399798176, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:38,184 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:38,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54892 deadline: 1679399798176, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:38,200 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:38,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54864 deadline: 1679399798176, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:38,201 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:38,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54904 deadline: 1679399798178, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:38,201 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:38,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54894 deadline: 1679399798178, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:38,204 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/dd688aa38f3845188d8c91c8f59da958, entries=150, sequenceid=406, filesize=11.9 K 2023-03-21 11:55:38,207 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~138.65 KB/141980, heapSize ~391.44 KB/400832, currentSize=46.96 KB/48090 for 1baade11f622687e2c1237b8441afd49 in 416ms, sequenceid=406, compaction requested=false 2023-03-21 11:55:38,211 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1baade11f622687e2c1237b8441afd49: 2023-03-21 11:55:38,216 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:55:38,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:55:38,234 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:55:38,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:55:38,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:55:38,252 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:55:38,252 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:38,254 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:55:38,255 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:55:38,256 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:38,256 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:38,258 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:38,258 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:38,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:38,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:38,259 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:38,259 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:55:38,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:38,260 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:38,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:55:38,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:38,260 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:55:38,266 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:55:38,266 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:55:38,266 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:55:38,266 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:38,267 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:55:38,270 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:55:38,270 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:38,270 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:55:38,270 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. started... 2023-03-21 11:55:38,270 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1baade11f622687e2c1237b8441afd49 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-21 11:55:38,271 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1baade11f622687e2c1237b8441afd49, store=A 2023-03-21 11:55:38,271 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:55:38,271 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1baade11f622687e2c1237b8441afd49, store=B 2023-03-21 11:55:38,271 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:55:38,271 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1baade11f622687e2c1237b8441afd49, store=C 2023-03-21 11:55:38,271 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:55:38,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(9158): Flush requested on 1baade11f622687e2c1237b8441afd49 2023-03-21 11:55:38,491 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. as already flushing 2023-03-21 11:55:38,583 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9554, totalHeapSize=21976, min timestamp=1679399738531, max timestamp=1679399738549Num uniques 859; 2023-03-21 11:55:38,583 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679399738531, max timestamp=1679399738549Num uniques 859; 2023-03-21 11:55:38,583 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-03-21 11:55:38,583 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9561, totalHeapSize=21976, min timestamp=1679399738531, max timestamp=1679399738549Num uniques 859; 2023-03-21 11:55:38,584 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=39331, new segment data size=9561, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-21 11:55:38,583 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=39324, new segment data size=9554, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-21 11:55:38,669 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:38,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54892 deadline: 1679399798627, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:38,674 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:38,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54864 deadline: 1679399798649, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:38,680 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:38,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54904 deadline: 1679399798671, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:38,683 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:38,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54894 deadline: 1679399798675, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:38,684 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:38,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54840 deadline: 1679399798678, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:38,748 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=416 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/A/428d7d3148644c638ab53ce30e985317 2023-03-21 11:55:38,783 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:38,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54892 deadline: 1679399798775, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:38,784 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:38,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54864 deadline: 1679399798776, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:38,788 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:38,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54904 deadline: 1679399798783, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:38,792 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:38,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54894 deadline: 1679399798785, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:38,793 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:38,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54840 deadline: 1679399798786, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:38,833 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=416 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/B/7ce5a963f01b477e988bd7787b392c48 2023-03-21 11:55:38,894 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=416 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/C/37682e8f339d4314b6b8a02999b5c9fd 2023-03-21 11:55:38,910 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/A/428d7d3148644c638ab53ce30e985317 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/428d7d3148644c638ab53ce30e985317 2023-03-21 11:55:38,933 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/428d7d3148644c638ab53ce30e985317, entries=150, sequenceid=416, filesize=11.9 K 2023-03-21 11:55:38,935 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/B/7ce5a963f01b477e988bd7787b392c48 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/7ce5a963f01b477e988bd7787b392c48 2023-03-21 11:55:38,954 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/7ce5a963f01b477e988bd7787b392c48, entries=150, sequenceid=416, filesize=11.9 K 2023-03-21 11:55:38,959 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/C/37682e8f339d4314b6b8a02999b5c9fd as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/37682e8f339d4314b6b8a02999b5c9fd 2023-03-21 11:55:38,978 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/37682e8f339d4314b6b8a02999b5c9fd, entries=150, sequenceid=416, filesize=11.9 K 2023-03-21 11:55:38,981 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=131.55 KB/134706 for 1baade11f622687e2c1237b8441afd49 in 711ms, sequenceid=416, compaction requested=true 2023-03-21 11:55:38,981 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1baade11f622687e2c1237b8441afd49: 2023-03-21 11:55:38,981 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:38,981 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:55:38,982 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:55:38,982 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:38,982 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:55:38,982 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:55:38,985 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:38,985 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:38,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:38,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:38,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:38,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:38,986 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:38,986 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:55:38,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:38,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:38,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:38,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:38,988 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:55:38,988 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d1a1c65[Count = 0] remaining members to acquire global barrier 2023-03-21 11:55:38,988 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:55:38,989 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:38,990 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:38,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:38,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:38,990 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:55:38,990 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:55:38,990 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:55:38,993 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:38,993 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:55:38,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(9158): Flush requested on 1baade11f622687e2c1237b8441afd49 2023-03-21 11:55:38,995 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1baade11f622687e2c1237b8441afd49 3/3 column families, dataSize=138.26 KB heapSize=394.23 KB 2023-03-21 11:55:38,997 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:55:38,997 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:38,997 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:39,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:39,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:39,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:39,002 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679399738630, max timestamp=1679399738994Num uniques 1009; 2023-03-21 11:55:39,002 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1679399738630, max timestamp=1679399738994Num uniques 1068; 2023-03-21 11:55:39,000 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1baade11f622687e2c1237b8441afd49, store=A 2023-03-21 11:55:39,000 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1679399738630, max timestamp=1679399738994Num uniques 1068; 2023-03-21 11:55:39,003 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-03-21 11:55:39,003 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-21 11:55:38,997 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:55:39,004 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1baade11f622687e2c1237b8441afd49, store=B 2023-03-21 11:55:39,003 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-03-21 11:55:39,003 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=6870, suffix heap size=132904,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=150 2023-03-21 11:55:39,004 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-21 11:55:39,004 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1baade11f622687e2c1237b8441afd49, store=C 2023-03-21 11:55:39,004 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-21 11:55:39,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:39,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:39,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:39,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:39,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:39,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:39,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:39,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:55:39,009 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:55:39,009 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:55:39,009 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:55:39,009 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:55:39,009 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:55:39,009 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:55:39,011 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:39,011 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:39,011 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:39,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:39,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:39,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:39,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:39,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:39,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:39,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:39,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:39,013 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:39,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:39,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:39,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:39,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:39,016 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:39,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:39,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:39,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:39,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:39,024 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:39,025 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:39,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:39,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:39,025 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:39,025 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:55:39,025 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:55:39,025 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:55:39,025 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:55:39,025 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:39,026 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:39,026 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:55:39,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:39,026 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:39,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:39,027 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:39,027 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:39,027 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:39,027 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:39,045 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679399739017, max timestamp=1679399739041Num uniques 859; 2023-03-21 11:55:39,045 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679399739017, max timestamp=1679399739041Num uniques 859; 2023-03-21 11:55:39,045 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-21 11:55:39,045 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-21 11:55:39,047 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679399739017, max timestamp=1679399739041Num uniques 859; 2023-03-21 11:55:39,047 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-21 11:55:39,069 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=454 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/A/116e34b7be594bb88be70982db0eec91 2023-03-21 11:55:39,097 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1679399739072, max timestamp=1679399739093Num uniques 1068; 2023-03-21 11:55:39,097 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1679399739072, max timestamp=1679399739093Num uniques 1068; 2023-03-21 11:55:39,097 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=9978, suffix heap size=139040,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=218 2023-03-21 11:55:39,097 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=9978, suffix heap size=139040,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=218 2023-03-21 11:55:39,100 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1679399739072, max timestamp=1679399739094Num uniques 1068; 2023-03-21 11:55:39,100 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-03-21 11:55:39,116 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=454 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/B/dca0897dc2be49dc98dd161927497655 2023-03-21 11:55:39,127 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:55:39,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:55:39,132 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:55:39,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:55:39,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:55:39,139 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:55:39,139 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:39,140 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:55:39,140 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:55:39,140 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:39,140 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:39,141 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:39,141 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:39,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:39,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:39,142 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:39,142 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:55:39,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:39,143 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:39,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:55:39,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:39,145 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:55:39,148 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:55:39,148 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:55:39,148 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:55:39,148 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:39,150 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1679399739118, max timestamp=1679399739145Num uniques 1077; 2023-03-21 11:55:39,151 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:55:39,151 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=8102, suffix heap size=139976,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=177 2023-03-21 11:55:39,153 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1679399739118, max timestamp=1679399739145Num uniques 1077; 2023-03-21 11:55:39,153 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=8102, suffix heap size=139976,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=177 2023-03-21 11:55:39,155 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8098, totalHeapSize=18648, min timestamp=1679399739118, max timestamp=1679399739145Num uniques 1027; 2023-03-21 11:55:39,155 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=47028, new segment data size=8098, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-03-21 11:55:39,156 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:55:39,157 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:39,157 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:55:39,158 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. started... 2023-03-21 11:55:39,158 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. as already flushing 2023-03-21 11:55:39,159 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:39,160 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:55:39,160 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:55:39,160 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:39,160 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:55:39,160 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:55:39,163 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:39,163 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:39,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:39,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:39,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:39,163 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:39,163 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:55:39,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:39,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:39,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:39,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:39,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:39,166 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:55:39,166 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5705071e[Count = 0] remaining members to acquire global barrier 2023-03-21 11:55:39,166 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:55:39,166 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:39,167 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:39,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:39,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:39,167 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:55:39,167 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:55:39,168 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:55:39,168 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:39,168 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:55:39,170 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:55:39,170 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:39,170 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:39,170 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:55:39,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:39,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:39,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:39,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:39,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:39,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:39,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:39,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:39,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:39,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:39,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:55:39,176 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:55:39,176 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:55:39,176 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:55:39,176 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:55:39,176 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:55:39,176 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:55:39,181 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:39,182 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:39,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:39,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:39,182 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:39,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:39,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:39,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:39,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:39,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:39,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:39,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:39,184 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:39,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:39,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:39,193 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:39,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:39,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:39,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:39,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:39,200 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1679399739179, max timestamp=1679399739194Num uniques 1036; 2023-03-21 11:55:39,200 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=10806, suffix heap size=135712,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=236 2023-03-21 11:55:39,200 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1679399739179, max timestamp=1679399739195Num uniques 1036; 2023-03-21 11:55:39,200 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=10806, suffix heap size=135712,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=236 2023-03-21 11:55:39,200 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1679399739179, max timestamp=1679399739194Num uniques 1036; 2023-03-21 11:55:39,200 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=10806, suffix heap size=135712,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=236 2023-03-21 11:55:39,200 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:39,200 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:39,200 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:39,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:39,200 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:39,200 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:39,200 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:39,201 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:39,201 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:55:39,201 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:55:39,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:39,202 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:55:39,202 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:55:39,202 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:55:39,201 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:39,202 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:39,202 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:39,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:39,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:39,239 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=345, cellSize=15800, totalHeapSize=36120, min timestamp=1679399739226, max timestamp=1679399739237Num uniques 1095; 2023-03-21 11:55:39,239 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=15800, suffix heap size=141848,new segment heap size=36120 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=345 2023-03-21 11:55:39,239 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=345, cellSize=15800, totalHeapSize=36120, min timestamp=1679399739226, max timestamp=1679399739237Num uniques 1095; 2023-03-21 11:55:39,240 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=15800, suffix heap size=141848,new segment heap size=36120 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=345 2023-03-21 11:55:39,240 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=345, cellSize=15800, totalHeapSize=36120, min timestamp=1679399739226, max timestamp=1679399739237Num uniques 1095; 2023-03-21 11:55:39,240 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=15800, suffix heap size=141848,new segment heap size=36120 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=345 2023-03-21 11:55:39,272 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:39,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54864 deadline: 1679399799262, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:39,273 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:39,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54840 deadline: 1679399799262, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:39,273 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11630, totalHeapSize=26656, min timestamp=1679399739252, max timestamp=1679399739264Num uniques 1204; 2023-03-21 11:55:39,273 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11630, totalHeapSize=26656, min timestamp=1679399739252, max timestamp=1679399739264Num uniques 1204; 2023-03-21 11:55:39,273 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=55140, new segment data size=11630, suffix heap size=153184,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1204, new segment cells count=254 2023-03-21 11:55:39,273 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11630, totalHeapSize=26656, min timestamp=1679399739252, max timestamp=1679399739264Num uniques 1204; 2023-03-21 11:55:39,273 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=55140, new segment data size=11630, suffix heap size=153184,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1204, new segment cells count=254 2023-03-21 11:55:39,274 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=55140, new segment data size=11630, suffix heap size=153184,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1204, new segment cells count=254 2023-03-21 11:55:39,307 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:55:39,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:55:39,317 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:55:39,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:55:39,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:55:39,320 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:55:39,320 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:39,322 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:55:39,323 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:55:39,323 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:39,323 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:39,324 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:39,324 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:39,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:39,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:39,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:39,325 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:39,326 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:55:39,326 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:39,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:55:39,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:39,326 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:55:39,327 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:55:39,327 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:55:39,327 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:55:39,327 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:39,330 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:55:39,331 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:55:39,331 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:55:39,331 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:39,331 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. started... 2023-03-21 11:55:39,331 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. as already flushing 2023-03-21 11:55:39,331 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:39,332 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:55:39,332 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:55:39,332 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:39,332 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:55:39,332 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:55:39,339 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:39,339 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:39,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:39,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:39,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:39,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:39,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:39,341 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:39,341 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:55:39,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:39,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:39,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:39,344 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:55:39,344 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@53f3fc2d[Count = 0] remaining members to acquire global barrier 2023-03-21 11:55:39,344 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:55:39,344 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:39,348 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:39,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:39,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:39,348 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:55:39,348 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:55:39,348 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:55:39,348 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:39,348 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:55:39,350 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:55:39,350 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:39,350 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:55:39,350 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:39,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:39,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:39,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:39,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:39,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:39,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:39,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:39,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:39,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:39,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:39,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:55:39,356 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:55:39,356 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:55:39,356 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:55:39,356 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:55:39,356 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:55:39,356 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:55:39,357 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=263, cellSize=12038, totalHeapSize=27592, min timestamp=1679399739323, max timestamp=1679399739354Num uniques 1113; 2023-03-21 11:55:39,357 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=50968, new segment data size=12038, suffix heap size=143720,new segment heap size=27592 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1113, new segment cells count=263 2023-03-21 11:55:39,359 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=263, cellSize=12038, totalHeapSize=27592, min timestamp=1679399739323, max timestamp=1679399739354Num uniques 1113; 2023-03-21 11:55:39,359 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=263, cellSize=12038, totalHeapSize=27592, min timestamp=1679399739323, max timestamp=1679399739354Num uniques 1113; 2023-03-21 11:55:39,359 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=50968, new segment data size=12038, suffix heap size=143720,new segment heap size=27592 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1113, new segment cells count=263 2023-03-21 11:55:39,359 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=50968, new segment data size=12038, suffix heap size=143720,new segment heap size=27592 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1113, new segment cells count=263 2023-03-21 11:55:39,364 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:39,364 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:39,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:39,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:39,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:39,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:39,364 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:39,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:39,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:39,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:39,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:39,384 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:39,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:39,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:39,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:39,391 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:39,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:39,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:39,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:39,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:39,395 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:39,395 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:39,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:39,395 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:39,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:39,395 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:39,396 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:55:39,396 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:55:39,396 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:39,396 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:39,396 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:39,396 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:55:39,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:39,396 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:55:39,396 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:55:39,396 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:39,396 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:39,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:39,397 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:39,419 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=322, cellSize=14742, totalHeapSize=33728, min timestamp=1679399739401, max timestamp=1679399739416Num uniques 1122; 2023-03-21 11:55:39,419 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=51382, new segment data size=14742, suffix heap size=144656,new segment heap size=33728 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1122, new segment cells count=322 2023-03-21 11:55:39,421 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1679399739402, max timestamp=1679399739418Num uniques 1122; 2023-03-21 11:55:39,421 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=51382, new segment data size=7872, suffix heap size=144656,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1122, new segment cells count=172 2023-03-21 11:55:39,423 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1679399739402, max timestamp=1679399739418Num uniques 1122; 2023-03-21 11:55:39,423 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=51382, new segment data size=7872, suffix heap size=144656,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1122, new segment cells count=172 2023-03-21 11:55:39,452 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=281, cellSize=12866, totalHeapSize=29464, min timestamp=1679399739433, max timestamp=1679399739449Num uniques 1181; 2023-03-21 11:55:39,452 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=281, cellSize=12866, totalHeapSize=29464, min timestamp=1679399739433, max timestamp=1679399739449Num uniques 1031; 2023-03-21 11:55:39,452 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=281, cellSize=12866, totalHeapSize=29464, min timestamp=1679399739433, max timestamp=1679399739449Num uniques 1031; 2023-03-21 11:55:39,452 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=12866, suffix heap size=135192,new segment heap size=29464 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=281 2023-03-21 11:55:39,452 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=54086, new segment data size=12866, suffix heap size=150792,new segment heap size=29464 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1181, new segment cells count=281 2023-03-21 11:55:39,454 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=12866, suffix heap size=135192,new segment heap size=29464 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=281 2023-03-21 11:55:39,483 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1679399739473, max timestamp=1679399739479Num uniques 1140; 2023-03-21 11:55:39,483 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=52210, new segment data size=8700, suffix heap size=146528,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1140, new segment cells count=190 2023-03-21 11:55:39,485 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1679399739473, max timestamp=1679399739479Num uniques 1140; 2023-03-21 11:55:39,485 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=52210, new segment data size=8700, suffix heap size=146528,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1140, new segment cells count=190 2023-03-21 11:55:39,485 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1679399739473, max timestamp=1679399739479Num uniques 1140; 2023-03-21 11:55:39,485 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=52210, new segment data size=8700, suffix heap size=146528,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1140, new segment cells count=190 2023-03-21 11:55:39,496 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:55:39,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:55:39,501 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:55:39,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:55:39,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:55:39,514 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:55:39,514 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:39,520 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:55:39,520 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:55:39,520 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:39,521 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:39,522 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:39,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:39,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:39,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:39,523 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:39,523 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:39,523 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:39,523 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:55:39,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:55:39,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:39,524 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:55:39,524 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:55:39,524 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:55:39,524 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:39,525 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:55:39,527 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:55:39,527 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:55:39,527 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:39,527 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. started... 2023-03-21 11:55:39,527 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. as already flushing 2023-03-21 11:55:39,527 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:39,527 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9104, totalHeapSize=20936, min timestamp=1679399739483, max timestamp=1679399739522Num uniques 1049; 2023-03-21 11:55:39,528 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:55:39,528 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:55:39,528 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:39,528 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:55:39,528 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:55:39,528 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679399739483, max timestamp=1679399739522Num uniques 1049; 2023-03-21 11:55:39,528 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=48034, new segment data size=9104, suffix heap size=137064,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=199 2023-03-21 11:55:39,528 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=48024, new segment data size=6870, suffix heap size=137064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=150 2023-03-21 11:55:39,528 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679399739483, max timestamp=1679399739522Num uniques 1049; 2023-03-21 11:55:39,528 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=48024, new segment data size=6870, suffix heap size=137064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=150 2023-03-21 11:55:39,532 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:39,532 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:39,532 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:55:39,534 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:39,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:39,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:39,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:39,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:39,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:39,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:39,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:39,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:39,537 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:55:39,537 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ccd0ec[Count = 0] remaining members to acquire global barrier 2023-03-21 11:55:39,537 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:55:39,537 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:39,548 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:39,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:39,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:39,548 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:55:39,548 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:55:39,548 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:55:39,548 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:39,549 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:55:39,550 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:39,550 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:55:39,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:39,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:39,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:39,550 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:39,550 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:55:39,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:39,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:39,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:39,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:39,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:39,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:39,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:39,561 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.15 KB at sequenceid=454 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/C/9dc2aed25ace42218915e42bd296a96f 2023-03-21 11:55:39,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:55:39,564 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:55:39,564 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:55:39,564 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:55:39,564 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:55:39,564 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:55:39,564 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:55:39,570 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=258, cellSize=11810, totalHeapSize=27072, min timestamp=1679399739547, max timestamp=1679399739567Num uniques 1058; 2023-03-21 11:55:39,570 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=258, cellSize=11810, totalHeapSize=27072, min timestamp=1679399739547, max timestamp=1679399739567Num uniques 1009; 2023-03-21 11:55:39,571 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=46226, new segment data size=11810, suffix heap size=132904,new segment heap size=27072 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=258 2023-03-21 11:55:39,570 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=11810, suffix heap size=138000,new segment heap size=27072 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=258 2023-03-21 11:55:39,571 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=258, cellSize=11810, totalHeapSize=27072, min timestamp=1679399739547, max timestamp=1679399739567Num uniques 1009; 2023-03-21 11:55:39,571 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=46226, new segment data size=11810, suffix heap size=132904,new segment heap size=27072 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=258 2023-03-21 11:55:39,574 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:39,574 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:39,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:39,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:39,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:39,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:39,576 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:39,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:39,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:39,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:39,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:39,577 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/A/116e34b7be594bb88be70982db0eec91 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/116e34b7be594bb88be70982db0eec91 2023-03-21 11:55:39,578 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:39,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:39,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:39,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:39,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:39,581 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:39,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:39,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:39,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:39,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:39,589 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:39,589 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:39,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:39,590 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:55:39,590 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:39,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:39,590 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:39,590 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:55:39,594 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:55:39,590 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:39,595 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:39,595 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:39,595 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:39,595 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:39,595 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:39,595 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:55:39,595 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:55:39,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:39,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:39,595 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/116e34b7be594bb88be70982db0eec91, entries=150, sequenceid=454, filesize=11.9 K 2023-03-21 11:55:39,599 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/B/dca0897dc2be49dc98dd161927497655 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/dca0897dc2be49dc98dd161927497655 2023-03-21 11:55:39,609 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/dca0897dc2be49dc98dd161927497655, entries=150, sequenceid=454, filesize=11.9 K 2023-03-21 11:55:39,616 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/C/9dc2aed25ace42218915e42bd296a96f as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/9dc2aed25ace42218915e42bd296a96f 2023-03-21 11:55:39,625 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:39,625 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=167, cellSize=7642, totalHeapSize=17608, min timestamp=1679399739597, max timestamp=1679399739618Num uniques 1117; 2023-03-21 11:55:39,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54892 deadline: 1679399799616, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:39,627 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=167, cellSize=7642, totalHeapSize=17608, min timestamp=1679399739597, max timestamp=1679399739618Num uniques 1117; 2023-03-21 11:55:39,627 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=51152, new segment data size=7642, suffix heap size=144136,new segment heap size=17608 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1117, new segment cells count=167 2023-03-21 11:55:39,627 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/9dc2aed25ace42218915e42bd296a96f, entries=150, sequenceid=454, filesize=11.9 K 2023-03-21 11:55:39,629 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~26.04 KB/26662, heapSize ~63.52 KB/65040, currentSize=89.45 KB/91600 for 1baade11f622687e2c1237b8441afd49 in 633ms, sequenceid=454, compaction requested=true 2023-03-21 11:55:39,629 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1baade11f622687e2c1237b8441afd49: 2023-03-21 11:55:39,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(9158): Flush requested on 1baade11f622687e2c1237b8441afd49 2023-03-21 11:55:39,629 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679399739597, max timestamp=1679399739618Num uniques 1117; 2023-03-21 11:55:39,629 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=51152, new segment data size=6870, suffix heap size=144136,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1117, new segment cells count=150 2023-03-21 11:55:39,625 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=51152, new segment data size=7642, suffix heap size=144136,new segment heap size=17608 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1117, new segment cells count=167 2023-03-21 11:55:39,639 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-21 11:55:39,639 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-21 11:55:39,640 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-21 11:55:39,640 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-21 11:55:39,640 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-21 11:55:39,643 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1baade11f622687e2c1237b8441afd49 3/3 column families, dataSize=52.92 KB heapSize=143.45 KB 2023-03-21 11:55:39,643 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1baade11f622687e2c1237b8441afd49, store=A 2023-03-21 11:55:39,644 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-21 11:55:39,644 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1baade11f622687e2c1237b8441afd49, store=B 2023-03-21 11:55:39,644 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-21 11:55:39,644 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1baade11f622687e2c1237b8441afd49, store=C 2023-03-21 11:55:39,644 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-21 11:55:39,655 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 46100 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-21 11:55:39,657 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1912): 1baade11f622687e2c1237b8441afd49/B is initiating minor compaction (all files) 2023-03-21 11:55:39,657 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1baade11f622687e2c1237b8441afd49/B in TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:39,658 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/0120925148ae4c879ca049603ff834e6, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/da21ab7ad4974bdf929c76397cb2653b, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/7ce5a963f01b477e988bd7787b392c48, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/dca0897dc2be49dc98dd161927497655] into tmpdir=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp, totalSize=45.0 K 2023-03-21 11:55:39,659 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 0120925148ae4c879ca049603ff834e6, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=9, earliestPutTs=1679399736488 2023-03-21 11:55:39,660 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 60744 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-21 11:55:39,660 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting da21ab7ad4974bdf929c76397cb2653b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=406, earliestPutTs=1679399737732 2023-03-21 11:55:39,673 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1912): 1baade11f622687e2c1237b8441afd49/A is initiating minor compaction (all files) 2023-03-21 11:55:39,673 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 1baade11f622687e2c1237b8441afd49/A in TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:39,673 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/1ab8e06c6cf6468db0b6a9f3d60d1717, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/779facedcefe4862b469363e0301cc77, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/428d7d3148644c638ab53ce30e985317, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/116e34b7be594bb88be70982db0eec91] into tmpdir=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp, totalSize=59.3 K 2023-03-21 11:55:39,676 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 1ab8e06c6cf6468db0b6a9f3d60d1717, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=9, earliestPutTs=1679399736488 2023-03-21 11:55:39,676 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 7ce5a963f01b477e988bd7787b392c48, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=416, earliestPutTs=1679399737799 2023-03-21 11:55:39,679 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting dca0897dc2be49dc98dd161927497655, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=454, earliestPutTs=1679399738630 2023-03-21 11:55:39,683 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 779facedcefe4862b469363e0301cc77, keycount=450, bloomtype=ROW, size=26.2 K, encoding=NONE, compression=NONE, seqNum=406, earliestPutTs=1679399737661 2023-03-21 11:55:39,694 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 428d7d3148644c638ab53ce30e985317, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=416, earliestPutTs=1679399737799 2023-03-21 11:55:39,695 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 116e34b7be594bb88be70982db0eec91, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=454, earliestPutTs=1679399738630 2023-03-21 11:55:39,696 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:55:39,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:55:39,700 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=409, cellSize=18725, totalHeapSize=42776, min timestamp=1679399739666, max timestamp=1679399739696Num uniques 859; 2023-03-21 11:55:39,700 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=18725, suffix heap size=117064,new segment heap size=42776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=409 2023-03-21 11:55:39,700 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:55:39,701 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=409, cellSize=18725, totalHeapSize=42776, min timestamp=1679399739666, max timestamp=1679399739696Num uniques 859; 2023-03-21 11:55:39,701 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=18725, suffix heap size=117064,new segment heap size=42776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=409 2023-03-21 11:55:39,701 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=409, cellSize=18725, totalHeapSize=42776, min timestamp=1679399739666, max timestamp=1679399739696Num uniques 859; 2023-03-21 11:55:39,701 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=18725, suffix heap size=117064,new segment heap size=42776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=409 2023-03-21 11:55:39,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:55:39,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:55:39,712 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:55:39,712 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:39,713 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:55:39,713 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:55:39,715 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:39,715 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:39,719 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=685 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/A/84b1b59a8e8d4ca8b78131c9edca2b4b 2023-03-21 11:55:39,719 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:39,719 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:39,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:39,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:39,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:39,720 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:39,720 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:55:39,720 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:39,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:55:39,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:39,721 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:55:39,730 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:55:39,730 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:55:39,730 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:55:39,730 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:39,731 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:55:39,732 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:55:39,732 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:55:39,732 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:39,732 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. started... 2023-03-21 11:55:39,732 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. as already flushing 2023-03-21 11:55:39,732 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:39,732 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:55:39,732 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:55:39,732 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:39,732 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:55:39,732 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:55:39,740 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:39,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:39,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:39,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:39,741 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:39,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:39,742 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:39,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:39,742 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:55:39,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:39,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:39,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:39,745 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:55:39,745 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6bd81d64[Count = 0] remaining members to acquire global barrier 2023-03-21 11:55:39,746 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:55:39,746 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:39,747 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:39,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54864 deadline: 1679399799721, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:39,747 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:39,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54904 deadline: 1679399799722, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:39,748 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:39,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54892 deadline: 1679399799742, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:39,749 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] throttle.PressureAwareThroughputController(145): 1baade11f622687e2c1237b8441afd49#B#compaction#13 average throughput is 0.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-21 11:55:39,749 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:39,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:39,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:39,750 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:55:39,750 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:55:39,750 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:55:39,750 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:39,750 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:55:39,751 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:55:39,751 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:39,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:39,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:39,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:39,751 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:39,752 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:55:39,751 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:39,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54840 deadline: 1679399799746, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:39,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:39,759 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] throttle.PressureAwareThroughputController(145): 1baade11f622687e2c1237b8441afd49#A#compaction#14 average throughput is 0.50 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-21 11:55:39,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:39,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:39,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:39,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:39,761 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:39,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:39,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:39,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54894 deadline: 1679399799747, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:39,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:55:39,764 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:55:39,764 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:55:39,764 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:55:39,764 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:55:39,764 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:55:39,764 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:55:39,767 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:39,767 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:39,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:39,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:39,767 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:39,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:39,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:39,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:39,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:39,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:39,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:39,768 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:39,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:39,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:39,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:39,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:39,770 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:39,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:39,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:39,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:39,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:39,781 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:39,781 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:39,781 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:55:39,781 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:55:39,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:39,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:39,782 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:55:39,782 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:55:39,781 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:39,781 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:39,781 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:39,781 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:55:39,782 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:39,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:39,782 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:39,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:39,782 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:39,782 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:39,782 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:39,814 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-03-21 11:55:39,852 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/A/081f2d6b58cc4ab59e75485f1a78c819 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/081f2d6b58cc4ab59e75485f1a78c819 2023-03-21 11:55:39,858 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:39,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54904 deadline: 1679399799849, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:39,859 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:39,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54864 deadline: 1679399799849, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:39,861 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:39,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54840 deadline: 1679399799861, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:39,881 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:39,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54894 deadline: 1679399799876, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:39,882 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:55:39,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:55:39,886 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:55:39,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:55:39,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:55:39,893 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:55:39,893 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:39,894 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:55:39,894 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:55:39,895 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:39,895 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:39,895 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:39,896 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:39,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:39,896 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 1baade11f622687e2c1237b8441afd49/A of 1baade11f622687e2c1237b8441afd49 into 081f2d6b58cc4ab59e75485f1a78c819(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-21 11:55:39,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:39,896 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1baade11f622687e2c1237b8441afd49: 2023-03-21 11:55:39,896 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:39,896 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:55:39,896 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49., storeName=1baade11f622687e2c1237b8441afd49/A, priority=12, startTime=1679399739635; duration=0sec 2023-03-21 11:55:39,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:39,896 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:39,897 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-21 11:55:39,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:55:39,897 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-21 11:55:39,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:39,897 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:55:39,897 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:55:39,899 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:55:39,900 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:55:39,900 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:39,900 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:55:39,901 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:55:39,901 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:39,901 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:55:39,901 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. started... 2023-03-21 11:55:39,901 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. as already flushing 2023-03-21 11:55:39,901 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:39,901 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:55:39,901 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:55:39,901 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:39,901 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:55:39,902 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:55:39,904 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:39,904 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:39,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:39,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:39,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:39,905 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:39,905 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:55:39,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:39,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:39,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:39,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:39,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:39,908 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:55:39,908 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@141acaf6[Count = 0] remaining members to acquire global barrier 2023-03-21 11:55:39,908 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:55:39,908 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:39,915 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:39,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:39,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:39,915 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:55:39,915 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:55:39,915 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:55:39,916 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:39,916 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:55:39,917 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 46100 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-21 11:55:39,917 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1912): 1baade11f622687e2c1237b8441afd49/C is initiating minor compaction (all files) 2023-03-21 11:55:39,917 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:39,917 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:55:39,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:39,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:39,917 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:39,917 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 1baade11f622687e2c1237b8441afd49/C in TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:39,917 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:55:39,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:39,917 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/d238a1367a424afcaa9c605131083402, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/dd688aa38f3845188d8c91c8f59da958, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/37682e8f339d4314b6b8a02999b5c9fd, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/9dc2aed25ace42218915e42bd296a96f] into tmpdir=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp, totalSize=45.0 K 2023-03-21 11:55:39,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:39,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:39,918 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting d238a1367a424afcaa9c605131083402, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=9, earliestPutTs=1679399736488 2023-03-21 11:55:39,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:39,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:39,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:39,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:39,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:39,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:55:39,923 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:55:39,924 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:55:39,924 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:55:39,924 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:55:39,924 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:55:39,924 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:55:39,927 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting dd688aa38f3845188d8c91c8f59da958, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=406, earliestPutTs=1679399737732 2023-03-21 11:55:39,931 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:39,931 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:39,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:39,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:39,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:39,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:39,931 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:39,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:39,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:39,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:39,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:39,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:39,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:39,932 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:39,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:39,933 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 37682e8f339d4314b6b8a02999b5c9fd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=416, earliestPutTs=1679399737799 2023-03-21 11:55:39,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:39,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:39,934 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 9dc2aed25ace42218915e42bd296a96f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=454, earliestPutTs=1679399738630 2023-03-21 11:55:39,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:39,935 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:39,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:39,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:39,940 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:39,940 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:39,940 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:39,940 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:39,940 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:39,940 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:39,940 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:39,941 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:39,941 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:39,941 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:39,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:39,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:39,942 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:55:39,942 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:55:39,942 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:55:39,942 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:55:39,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:39,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:39,961 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:39,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54892 deadline: 1679399799959, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:39,966 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-03-21 11:55:39,969 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-03-21 11:55:39,980 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] throttle.PressureAwareThroughputController(145): 1baade11f622687e2c1237b8441afd49#C#compaction#16 average throughput is 0.66 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-21 11:55:40,027 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/C/8c5c3e47186c4063b36028ad05769e5d as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/8c5c3e47186c4063b36028ad05769e5d 2023-03-21 11:55:40,038 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 1baade11f622687e2c1237b8441afd49/C of 1baade11f622687e2c1237b8441afd49 into 8c5c3e47186c4063b36028ad05769e5d(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-21 11:55:40,038 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1baade11f622687e2c1237b8441afd49: 2023-03-21 11:55:40,038 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49., storeName=1baade11f622687e2c1237b8441afd49/C, priority=12, startTime=1679399739639; duration=0sec 2023-03-21 11:55:40,038 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-21 11:55:40,043 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:55:40,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:55:40,050 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:55:40,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:55:40,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:55:40,052 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:55:40,052 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:40,056 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:55:40,056 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:55:40,056 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:40,057 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:40,057 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:40,057 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:40,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:40,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:40,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:40,059 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:40,059 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:40,059 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:55:40,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:55:40,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:40,059 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:55:40,060 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:55:40,060 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:55:40,060 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:55:40,060 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:40,061 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:55:40,061 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:55:40,061 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:55:40,061 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:40,061 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. started... 2023-03-21 11:55:40,061 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. as already flushing 2023-03-21 11:55:40,061 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:40,061 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:55:40,061 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:55:40,061 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:40,061 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:55:40,061 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:55:40,064 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:40,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54904 deadline: 1679399800062, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:40,065 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:40,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54864 deadline: 1679399800062, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:40,071 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:40,071 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:40,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:40,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:40,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:40,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:40,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:40,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:40,072 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:40,072 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:55:40,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:40,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:40,073 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:55:40,073 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@58444ecc[Count = 0] remaining members to acquire global barrier 2023-03-21 11:55:40,073 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:55:40,073 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:40,074 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:40,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:40,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:40,074 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:55:40,074 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:40,074 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:55:40,074 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:55:40,074 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:55:40,075 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:40,075 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:55:40,075 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:40,075 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:55:40,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:40,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:40,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:40,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:40,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:40,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:40,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:40,081 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:40,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54840 deadline: 1679399800078, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:40,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:40,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:40,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:40,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:55:40,085 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:55:40,085 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:55:40,085 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:55:40,085 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:55:40,085 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:55:40,085 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:55:40,086 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:40,086 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:40,086 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:40,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:40,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:40,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:40,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:40,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:40,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:40,087 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:40,087 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:40,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:40,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:40,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:40,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:40,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:40,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:40,088 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:40,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:40,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:40,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54894 deadline: 1679399800084, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:40,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:40,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:40,091 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:40,091 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:40,091 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:40,091 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:40,091 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:55:40,093 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:40,093 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:55:40,092 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:55:40,091 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:40,094 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:55:40,094 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:55:40,094 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:40,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:40,093 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:40,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:40,095 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:40,095 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:40,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:40,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:40,194 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:55:40,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:55:40,208 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:55:40,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:55:40,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:55:40,214 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:55:40,214 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:40,216 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:55:40,216 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:55:40,217 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:40,217 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:40,218 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:40,218 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:40,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:40,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:40,219 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:40,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:40,219 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:55:40,219 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:40,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:55:40,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:40,220 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:55:40,222 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:55:40,222 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:55:40,222 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:55:40,222 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:40,228 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:55:40,229 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:55:40,229 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:55:40,236 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=685 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/B/8446248109cd4bc484adc1e36995b117 2023-03-21 11:55:40,239 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:40,239 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. started... 2023-03-21 11:55:40,239 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. as already flushing 2023-03-21 11:55:40,239 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:40,239 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:55:40,239 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:55:40,239 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:40,239 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:55:40,239 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:55:40,241 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:40,241 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:40,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:40,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:40,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:40,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:40,242 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:40,242 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:55:40,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:40,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:40,243 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/B/ac41af5f10f0440bba4a366f5df13aeb as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/ac41af5f10f0440bba4a366f5df13aeb 2023-03-21 11:55:40,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:40,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:40,244 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:55:40,244 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@386fbe98[Count = 0] remaining members to acquire global barrier 2023-03-21 11:55:40,244 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:55:40,252 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:40,256 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:40,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:40,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:40,256 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:55:40,256 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:55:40,256 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:55:40,257 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:40,257 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:55:40,263 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:40,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:40,263 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:55:40,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:40,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:40,263 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:40,263 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:55:40,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:40,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:40,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:40,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:40,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:40,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:40,266 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 1baade11f622687e2c1237b8441afd49/B of 1baade11f622687e2c1237b8441afd49 into ac41af5f10f0440bba4a366f5df13aeb(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-21 11:55:40,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:40,266 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1baade11f622687e2c1237b8441afd49: 2023-03-21 11:55:40,266 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49., storeName=1baade11f622687e2c1237b8441afd49/B, priority=12, startTime=1679399739639; duration=0sec 2023-03-21 11:55:40,266 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-21 11:55:40,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:55:40,267 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:55:40,267 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:55:40,267 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:55:40,267 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:55:40,267 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:55:40,267 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:55:40,268 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:40,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54892 deadline: 1679399800264, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:40,274 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:40,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:40,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:40,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:40,274 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:40,275 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:40,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:40,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:40,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:40,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:40,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:40,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:40,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:40,278 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:40,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:40,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:40,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:40,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:40,280 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:40,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:40,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:40,282 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:40,282 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:40,282 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:40,282 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:40,282 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:40,282 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:40,282 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:40,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:40,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:40,282 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:40,282 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:40,282 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:55:40,282 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:55:40,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:40,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:40,283 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:55:40,283 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:55:40,282 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:40,368 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:40,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54904 deadline: 1679399800367, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:40,370 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:40,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54864 deadline: 1679399800368, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:40,383 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:55:40,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:55:40,384 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:40,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54840 deadline: 1679399800383, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:40,386 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:55:40,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:55:40,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:55:40,387 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:55:40,387 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:40,388 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:55:40,388 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:55:40,389 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:40,389 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:40,390 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:40,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:40,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:40,390 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:40,390 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:40,390 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:55:40,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:40,391 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:40,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:55:40,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:40,391 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:55:40,391 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:55:40,391 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:55:40,391 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:55:40,392 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:40,392 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:55:40,392 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:55:40,393 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:55:40,393 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:40,393 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. started... 2023-03-21 11:55:40,393 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. as already flushing 2023-03-21 11:55:40,393 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:40,393 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:55:40,393 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:55:40,393 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:40,393 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:55:40,393 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:55:40,394 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:40,394 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:40,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:40,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:40,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:40,395 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:40,395 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:55:40,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:40,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:40,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:40,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:40,396 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:40,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54894 deadline: 1679399800394, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:40,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:40,397 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:55:40,397 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1377b7cc[Count = 0] remaining members to acquire global barrier 2023-03-21 11:55:40,397 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:55:40,397 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:40,398 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:40,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:40,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:40,398 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:55:40,398 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:55:40,398 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:55:40,399 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:40,399 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:55:40,399 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:40,399 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:55:40,399 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:40,399 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:55:40,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:40,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:40,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:40,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:40,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:40,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:40,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:40,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:40,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:40,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:40,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:55:40,403 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:55:40,403 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:55:40,403 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:55:40,403 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:55:40,403 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:55:40,403 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:55:40,404 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:40,404 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:40,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:40,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:40,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:40,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:40,404 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:40,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:40,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:40,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:40,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:40,405 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:40,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:40,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:40,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:40,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:40,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:40,406 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:40,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:40,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:40,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:40,409 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:40,409 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:40,410 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:40,410 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:55:40,410 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:55:40,410 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:55:40,410 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:55:40,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:40,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:40,410 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:40,410 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:40,410 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:40,410 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:40,410 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:40,410 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:40,410 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:40,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:40,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:40,510 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:55:40,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:55:40,513 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:55:40,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:55:40,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:55:40,514 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:55:40,514 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:40,515 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:55:40,515 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:55:40,515 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:40,516 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:40,517 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:40,517 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:40,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:40,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:40,518 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:40,518 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:55:40,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:40,518 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:40,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:55:40,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:40,518 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:55:40,519 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:55:40,519 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:55:40,519 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:55:40,519 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:40,520 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:55:40,520 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:55:40,520 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:40,520 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:55:40,520 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. started... 2023-03-21 11:55:40,520 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. as already flushing 2023-03-21 11:55:40,520 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:40,520 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:55:40,520 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:55:40,520 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:40,520 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:55:40,520 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:55:40,521 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:40,521 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:40,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:40,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:40,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:40,522 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:40,522 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:55:40,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:40,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:40,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:40,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:40,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:40,524 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:55:40,524 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@51f8822e[Count = 0] remaining members to acquire global barrier 2023-03-21 11:55:40,524 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:55:40,524 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:40,524 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:40,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:40,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:40,525 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:55:40,525 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:40,525 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:55:40,525 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:55:40,525 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:55:40,526 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:55:40,526 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:40,526 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:40,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:40,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:40,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:40,526 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:55:40,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:40,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:40,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:40,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:40,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:40,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:40,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:40,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:55:40,530 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:55:40,530 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:55:40,530 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:55:40,530 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:55:40,530 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:55:40,530 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:55:40,531 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:40,531 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:40,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:40,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:40,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:40,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:40,531 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:40,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:40,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:40,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:40,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:40,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:40,534 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:40,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:40,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:40,536 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:40,537 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:40,537 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:40,537 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:40,537 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:40,538 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:40,538 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:40,538 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:40,538 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:40,538 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:40,538 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:55:40,539 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:55:40,539 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:55:40,539 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:55:40,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:40,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:40,538 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:40,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:40,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:40,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:40,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:40,548 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:55:40,639 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:55:40,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:55:40,643 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:55:40,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:55:40,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:55:40,645 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:55:40,645 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:40,646 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:55:40,646 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:55:40,646 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:40,647 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:40,647 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:40,647 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:40,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:40,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:40,648 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:40,648 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:55:40,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:40,648 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:40,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:55:40,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:40,649 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:55:40,649 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:55:40,649 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:55:40,649 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:55:40,649 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:40,650 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:55:40,651 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:55:40,651 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:55:40,651 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:40,651 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. started... 2023-03-21 11:55:40,651 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. as already flushing 2023-03-21 11:55:40,651 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:40,651 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:55:40,651 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:55:40,651 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:40,651 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:55:40,651 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:55:40,654 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:40,654 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:40,654 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:55:40,658 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:40,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:40,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:40,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:40,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:40,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:40,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:40,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:40,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:40,667 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:55:40,668 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2bb084f3[Count = 0] remaining members to acquire global barrier 2023-03-21 11:55:40,668 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:55:40,668 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:40,669 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:40,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:40,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:40,669 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:55:40,669 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:55:40,669 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:55:40,670 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:40,670 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:55:40,670 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:55:40,670 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:40,670 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:55:40,672 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:40,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:40,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:40,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:40,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:40,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:40,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:40,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:40,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:40,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:40,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:40,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:55:40,675 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:55:40,675 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:55:40,675 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:55:40,675 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:55:40,675 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:55:40,675 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:55:40,677 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:40,677 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:40,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:40,677 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:40,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:40,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:40,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:40,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:40,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:40,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:40,677 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:40,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:40,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:40,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:40,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:40,679 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:40,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:40,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:40,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:40,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:40,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:40,681 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:40,681 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:40,687 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:40,687 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:40,687 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:40,687 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:40,687 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:40,687 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:40,681 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:55:40,687 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:55:40,682 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:40,687 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:40,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:40,687 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:55:40,687 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:55:40,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:40,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:40,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:40,689 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:55:40,710 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.14 KB at sequenceid=685 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/C/d6252139523742b1ae6cc1cc4414ceca 2023-03-21 11:55:40,730 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/A/84b1b59a8e8d4ca8b78131c9edca2b4b as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/84b1b59a8e8d4ca8b78131c9edca2b4b 2023-03-21 11:55:40,753 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/84b1b59a8e8d4ca8b78131c9edca2b4b, entries=300, sequenceid=685, filesize=19.0 K 2023-03-21 11:55:40,755 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/B/8446248109cd4bc484adc1e36995b117 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/8446248109cd4bc484adc1e36995b117 2023-03-21 11:55:40,775 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/8446248109cd4bc484adc1e36995b117, entries=150, sequenceid=685, filesize=11.9 K 2023-03-21 11:55:40,786 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:40,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54892 deadline: 1679399800784, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:40,787 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/C/d6252139523742b1ae6cc1cc4414ceca as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/d6252139523742b1ae6cc1cc4414ceca 2023-03-21 11:55:40,788 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:55:40,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:55:40,792 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:55:40,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:55:40,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:55:40,803 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:55:40,803 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:40,805 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:55:40,805 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:55:40,805 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:40,806 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:40,806 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:40,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:40,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:40,807 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:40,807 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:40,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:40,807 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:55:40,808 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:40,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:55:40,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:40,808 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:55:40,810 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:55:40,810 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:55:40,810 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:55:40,810 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:40,811 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:55:40,812 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:55:40,812 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:55:40,812 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:40,812 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. started... 2023-03-21 11:55:40,812 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. as already flushing 2023-03-21 11:55:40,812 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:40,812 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:55:40,812 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:55:40,812 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:40,812 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:55:40,812 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:55:40,814 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/d6252139523742b1ae6cc1cc4414ceca, entries=150, sequenceid=685, filesize=11.9 K 2023-03-21 11:55:40,814 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:40,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:40,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:40,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:40,815 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:40,815 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:40,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:40,815 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:55:40,816 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~52.92 KB/54188, heapSize ~143.40 KB/146840, currentSize=140.89 KB/144270 for 1baade11f622687e2c1237b8441afd49 in 1175ms, sequenceid=685, compaction requested=false 2023-03-21 11:55:40,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:40,816 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1baade11f622687e2c1237b8441afd49: 2023-03-21 11:55:40,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:40,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:40,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:40,818 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:55:40,818 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e6e7463[Count = 0] remaining members to acquire global barrier 2023-03-21 11:55:40,818 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:55:40,818 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:40,823 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:40,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:40,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:40,823 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:55:40,823 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:40,823 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:55:40,823 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:55:40,823 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:55:40,825 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:55:40,825 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:40,825 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:55:40,825 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:40,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:40,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:40,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:40,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:40,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:40,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:40,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:40,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:40,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:40,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:40,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:55:40,830 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:55:40,830 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:55:40,830 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:55:40,830 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:55:40,830 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:55:40,830 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:55:40,838 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:40,838 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:40,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:40,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:40,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:40,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:40,838 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:40,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:40,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:40,838 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:40,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:40,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:40,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:40,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:40,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:40,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:40,840 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:40,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:40,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:40,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:40,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:40,846 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:40,846 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:40,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:40,846 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:40,846 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:55:40,846 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:40,846 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:55:40,847 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:55:40,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:40,846 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:40,847 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:40,847 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:40,847 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:40,847 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:40,847 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:40,847 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:55:40,847 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:55:40,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:40,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:40,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(9158): Flush requested on 1baade11f622687e2c1237b8441afd49 2023-03-21 11:55:40,874 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1baade11f622687e2c1237b8441afd49 3/3 column families, dataSize=147.60 KB heapSize=401.39 KB 2023-03-21 11:55:40,875 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1baade11f622687e2c1237b8441afd49, store=A 2023-03-21 11:55:40,875 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-21 11:55:40,875 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1baade11f622687e2c1237b8441afd49, store=B 2023-03-21 11:55:40,875 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-21 11:55:40,875 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1baade11f622687e2c1237b8441afd49, store=C 2023-03-21 11:55:40,875 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-21 11:55:40,896 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:40,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54864 deadline: 1679399800888, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:40,899 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:40,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54840 deadline: 1679399800890, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:40,900 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:40,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54904 deadline: 1679399800894, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:40,903 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:40,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54894 deadline: 1679399800902, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:40,948 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:55:40,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:55:40,950 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:55:40,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:55:40,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:55:40,952 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:55:40,952 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:40,952 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:55:40,953 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:55:40,953 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:40,953 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:40,954 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:40,954 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:40,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:40,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:40,954 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:40,954 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:55:40,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:40,954 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:40,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:55:40,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:40,955 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:55:40,955 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:55:40,955 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:55:40,955 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:55:40,955 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:40,956 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:55:40,956 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:55:40,956 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:55:40,956 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:40,956 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. started... 2023-03-21 11:55:40,956 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. as already flushing 2023-03-21 11:55:40,956 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:40,956 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:55:40,956 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:55:40,956 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:40,956 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:55:40,956 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:55:40,957 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:40,957 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:40,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:40,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:40,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:40,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:40,958 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:40,958 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:55:40,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:40,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:40,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:40,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:40,959 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:55:40,959 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@400977b5[Count = 0] remaining members to acquire global barrier 2023-03-21 11:55:40,959 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:55:40,959 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:40,960 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:40,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:40,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:40,960 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:55:40,960 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:55:40,961 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:55:40,961 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:40,961 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:55:40,961 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:40,961 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:55:40,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:40,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:40,962 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:40,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:40,962 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:55:40,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:40,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:40,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:40,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:40,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:40,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:40,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:40,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:55:40,965 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:55:40,965 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:55:40,965 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:55:40,965 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:55:40,965 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:55:40,965 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:55:40,966 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:40,966 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:40,966 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:40,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:40,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:40,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:40,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:40,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:40,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:40,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:40,967 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:40,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:40,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:40,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:40,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:40,968 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:40,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:40,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:40,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:40,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:40,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:40,970 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:40,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:40,971 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:40,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:40,971 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:40,971 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:40,971 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:40,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:40,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:40,971 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:40,971 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:40,971 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:40,971 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:55:40,971 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:55:40,971 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:40,971 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:40,972 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:55:40,974 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:55:40,975 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:55:41,003 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:41,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54864 deadline: 1679399801002, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:41,003 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:41,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54904 deadline: 1679399801002, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:41,075 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:55:41,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:55:41,078 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:55:41,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:55:41,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:55:41,079 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:55:41,079 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:41,080 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:55:41,080 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:55:41,081 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:41,082 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:41,082 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:41,082 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:41,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:41,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:41,083 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:41,083 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:55:41,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:41,083 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:41,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:55:41,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:41,083 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:55:41,084 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:55:41,084 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:55:41,084 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:55:41,084 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:41,085 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:55:41,085 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:55:41,085 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:55:41,085 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:41,085 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. started... 2023-03-21 11:55:41,085 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. as already flushing 2023-03-21 11:55:41,085 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:41,086 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:55:41,086 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:55:41,086 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:41,086 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:55:41,086 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:55:41,087 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:41,087 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:41,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:41,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:41,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:41,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:41,087 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:41,088 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:55:41,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:41,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:41,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:41,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:41,089 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:55:41,089 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@55f56468[Count = 0] remaining members to acquire global barrier 2023-03-21 11:55:41,089 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:55:41,089 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:41,090 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:41,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:41,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:41,090 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:55:41,090 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:55:41,090 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:55:41,090 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:41,091 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:55:41,091 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:55:41,091 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:41,091 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:55:41,092 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:41,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:41,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:41,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:41,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:41,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:41,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:41,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:41,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:41,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:41,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:41,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:55:41,095 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:55:41,095 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:55:41,095 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:55:41,095 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:55:41,095 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:55:41,095 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:55:41,097 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:41,097 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:41,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:41,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:41,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:41,098 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:41,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:41,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:41,098 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:41,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:41,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:41,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:41,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:41,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:41,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:41,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:41,099 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:41,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:41,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:41,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:41,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:41,101 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:41,101 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:41,101 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:41,101 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:41,101 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:41,101 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:41,101 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:41,101 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:55:41,102 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:41,102 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:41,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:41,102 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:55:41,102 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:55:41,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:41,102 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:41,103 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:55:41,103 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:55:41,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:41,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:41,203 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:55:41,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:55:41,206 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:55:41,206 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:41,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54904 deadline: 1679399801206, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:41,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:55:41,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:55:41,207 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:55:41,208 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:41,208 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:41,208 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:55:41,209 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:55:41,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54864 deadline: 1679399801206, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:41,209 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:41,209 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:41,210 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:41,210 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:41,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:41,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:41,211 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:41,211 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:55:41,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:41,211 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:41,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:55:41,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:41,212 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:55:41,212 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:55:41,212 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:55:41,212 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:55:41,212 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:41,213 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:55:41,213 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:55:41,213 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:55:41,213 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:41,213 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. started... 2023-03-21 11:55:41,213 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. as already flushing 2023-03-21 11:55:41,214 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:41,214 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:55:41,214 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:55:41,214 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:41,214 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:55:41,214 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:55:41,215 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:41,215 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:41,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:41,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:41,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:41,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:41,215 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:41,215 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:55:41,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:41,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:41,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:41,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:41,219 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:55:41,219 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a4de539[Count = 0] remaining members to acquire global barrier 2023-03-21 11:55:41,219 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:55:41,219 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:41,223 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:41,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:41,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:41,223 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:55:41,223 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:55:41,223 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:55:41,224 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:41,224 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:55:41,225 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:55:41,225 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:41,225 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:41,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:41,228 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-03-21 11:55:41,225 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:55:41,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:41,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:41,229 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2023-03-21 11:55:41,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:41,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:41,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:41,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:41,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:41,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:41,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:41,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:55:41,232 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:55:41,232 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:55:41,232 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:55:41,232 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:55:41,232 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:55:41,232 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:55:41,239 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:41,239 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:41,239 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:41,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:41,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:41,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:41,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:41,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:41,240 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:41,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:41,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:41,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:41,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:41,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:41,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:41,241 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:41,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:41,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:41,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:41,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:41,248 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:41,248 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:41,248 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:41,248 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:41,248 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:41,248 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:41,248 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:41,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:41,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:41,248 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:55:41,249 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:55:41,248 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:41,250 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:41,250 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:41,250 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:55:41,250 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:55:41,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:41,250 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:55:41,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:41,302 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=723 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/A/e5f868adc3ec45018d376cfef4d45689 2023-03-21 11:55:41,334 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=723 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/B/bb9bdec46c2348a289d415ae05de9ae2 2023-03-21 11:55:41,350 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:55:41,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:55:41,357 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:55:41,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:55:41,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:55:41,358 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:55:41,358 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:41,359 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:55:41,359 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:55:41,360 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:41,360 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:41,360 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:41,361 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:41,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:41,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:41,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:41,361 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:41,361 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:55:41,361 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:41,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:55:41,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:41,362 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:55:41,362 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:55:41,362 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:55:41,364 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:55:41,364 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:41,364 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:55:41,365 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:55:41,365 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:55:41,365 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:41,365 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. started... 2023-03-21 11:55:41,365 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. as already flushing 2023-03-21 11:55:41,365 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:41,365 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:55:41,365 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:55:41,365 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:41,365 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:55:41,365 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:55:41,373 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:41,373 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:41,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:41,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:41,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:41,373 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:41,373 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:55:41,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:41,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:41,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:41,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:41,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:41,374 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:55:41,374 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@671d4b7f[Count = 0] remaining members to acquire global barrier 2023-03-21 11:55:41,374 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:55:41,375 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:41,379 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:41,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:41,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:41,380 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:55:41,380 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:55:41,380 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:55:41,381 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:41,381 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:55:41,382 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:55:41,382 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:41,382 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:41,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:41,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:41,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:41,383 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:55:41,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:41,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:41,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:41,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:41,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:41,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:41,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:41,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:55:41,387 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:55:41,387 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:55:41,387 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:55:41,387 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:55:41,387 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:55:41,387 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:55:41,390 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:41,390 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:41,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:41,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:41,390 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:41,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:41,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:41,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:41,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:41,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:41,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:41,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:41,390 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:41,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:41,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:41,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:41,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:41,392 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:41,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:41,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:41,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:41,396 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:41,396 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:41,396 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:41,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:41,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:41,396 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:41,396 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:55:41,396 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:55:41,396 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:41,396 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:41,399 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:41,399 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:41,399 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:41,399 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:41,399 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:55:41,399 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:55:41,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:41,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:41,499 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:55:41,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:55:41,502 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:55:41,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:55:41,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:55:41,504 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:55:41,504 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:41,504 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:55:41,505 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:55:41,505 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:41,505 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:41,506 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:41,506 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:41,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:41,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:41,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:41,507 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:41,507 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:55:41,507 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:41,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:55:41,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:41,507 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:55:41,507 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:55:41,508 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:55:41,508 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:55:41,508 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:41,508 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:55:41,509 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:55:41,509 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:55:41,509 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:41,509 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. started... 2023-03-21 11:55:41,509 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. as already flushing 2023-03-21 11:55:41,509 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:41,509 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:55:41,509 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:55:41,509 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:41,509 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:55:41,509 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:55:41,511 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:41,511 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:41,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:41,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:41,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:41,511 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:41,511 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:55:41,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:41,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:41,512 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:41,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54864 deadline: 1679399801512, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:41,513 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:41,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54904 deadline: 1679399801513, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:41,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:41,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:41,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:41,514 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:55:41,514 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ca23cf6[Count = 0] remaining members to acquire global barrier 2023-03-21 11:55:41,514 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:55:41,514 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:41,515 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:41,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:41,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:41,515 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:55:41,515 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:55:41,515 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:55:41,515 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:41,515 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:55:41,516 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:41,516 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:55:41,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:41,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:41,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:41,516 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:41,516 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:55:41,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:41,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:41,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:41,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:41,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:41,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:41,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:41,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:55:41,520 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:55:41,520 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:55:41,520 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:55:41,520 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:55:41,520 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:55:41,520 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:55:41,538 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:41,538 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:41,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:41,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:41,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:41,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:41,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:41,538 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:41,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:41,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:41,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:41,538 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:41,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:41,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:41,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:41,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:41,539 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:41,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:41,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:41,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:41,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:41,542 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:41,542 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:41,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:41,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:41,542 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:55:41,542 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:55:41,542 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:41,542 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:55:41,543 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:55:41,543 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:55:41,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:41,542 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:41,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:41,544 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:41,544 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:41,544 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:41,544 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:41,544 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:41,544 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:41,643 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:55:41,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:55:41,645 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:55:41,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:55:41,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:55:41,646 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:55:41,646 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:41,647 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:55:41,647 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:55:41,648 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:41,648 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:41,648 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:41,648 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:41,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:41,649 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:41,649 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:55:41,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:41,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:41,649 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:41,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:55:41,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:41,650 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:55:41,650 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:55:41,650 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:55:41,650 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:55:41,650 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:41,652 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:55:41,652 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:55:41,652 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:55:41,652 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:41,652 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. started... 2023-03-21 11:55:41,652 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. as already flushing 2023-03-21 11:55:41,652 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:41,652 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:55:41,652 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:55:41,652 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:41,652 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:55:41,652 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:55:41,653 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:41,653 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:41,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:41,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:41,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:41,654 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:41,654 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:55:41,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:41,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:41,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:41,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:41,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:41,655 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:55:41,655 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@128ad32e[Count = 0] remaining members to acquire global barrier 2023-03-21 11:55:41,655 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:55:41,655 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:41,656 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:41,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:41,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:41,656 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:55:41,656 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:41,656 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:55:41,656 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:55:41,656 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:55:41,657 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:41,657 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:55:41,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:41,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:41,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:41,657 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:41,657 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:55:41,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:41,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:41,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:41,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:41,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:41,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:41,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:41,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:55:41,659 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:55:41,659 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:55:41,659 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:55:41,659 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:55:41,659 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:55:41,659 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:55:41,661 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:41,661 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:41,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:41,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:41,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:41,661 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:41,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:41,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:41,661 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:41,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:41,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:41,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:41,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:41,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:41,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:41,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:41,663 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:41,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:41,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:41,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:41,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:41,666 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:41,666 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:41,666 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:41,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:41,666 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:55:41,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:41,666 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:55:41,666 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:55:41,666 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:41,666 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:41,666 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:41,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:41,668 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:55:41,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:41,668 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:41,669 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:55:41,669 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:41,669 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:41,669 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:41,756 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=723 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/C/24d0729fddcc4536a16ba5ed97d23127 2023-03-21 11:55:41,770 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:55:41,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:55:41,783 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/A/e5f868adc3ec45018d376cfef4d45689 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/e5f868adc3ec45018d376cfef4d45689 2023-03-21 11:55:41,784 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:55:41,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:55:41,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:55:41,787 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:55:41,787 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:41,787 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:55:41,787 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:55:41,789 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:41,789 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:41,790 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:41,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:41,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:41,790 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:41,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:41,791 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:41,791 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:55:41,791 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:41,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:55:41,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:41,791 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:55:41,792 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:55:41,792 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:55:41,792 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:41,792 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:55:41,792 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:55:41,792 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:55:41,792 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:41,792 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. started... 2023-03-21 11:55:41,792 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. as already flushing 2023-03-21 11:55:41,792 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:41,792 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:55:41,793 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:55:41,793 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:41,793 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:55:41,793 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:55:41,799 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:41,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54892 deadline: 1679399801797, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:41,801 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/e5f868adc3ec45018d376cfef4d45689, entries=250, sequenceid=723, filesize=16.7 K 2023-03-21 11:55:41,804 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:41,804 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:41,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:41,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:41,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:41,810 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:41,811 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:55:41,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:41,811 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/B/bb9bdec46c2348a289d415ae05de9ae2 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/bb9bdec46c2348a289d415ae05de9ae2 2023-03-21 11:55:41,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:41,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:41,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:41,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:41,818 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:55:41,818 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7fb986b[Count = 0] remaining members to acquire global barrier 2023-03-21 11:55:41,818 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:55:41,818 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:41,820 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:41,820 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:55:41,821 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:41,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:41,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:41,821 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:55:41,821 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:55:41,821 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:55:41,822 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:55:41,822 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:41,822 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:55:41,823 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:41,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:41,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:41,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:41,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:41,824 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/bb9bdec46c2348a289d415ae05de9ae2, entries=150, sequenceid=723, filesize=11.9 K 2023-03-21 11:55:41,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:41,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:41,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:41,830 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/C/24d0729fddcc4536a16ba5ed97d23127 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/24d0729fddcc4536a16ba5ed97d23127 2023-03-21 11:55:41,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:41,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:41,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:41,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:55:41,834 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:55:41,834 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:55:41,834 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:55:41,834 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:55:41,834 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:55:41,834 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:55:41,839 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:41,840 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:41,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:41,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:41,840 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:41,840 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/24d0729fddcc4536a16ba5ed97d23127, entries=150, sequenceid=723, filesize=11.9 K 2023-03-21 11:55:41,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:41,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:41,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:41,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:41,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:41,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:41,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:41,843 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~154.31 KB/158010, heapSize ~421.27 KB/431376, currentSize=33.54 KB/34350 for 1baade11f622687e2c1237b8441afd49 in 969ms, sequenceid=723, compaction requested=true 2023-03-21 11:55:41,843 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:41,843 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1baade11f622687e2c1237b8441afd49: 2023-03-21 11:55:41,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:41,844 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-21 11:55:41,844 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-21 11:55:41,844 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-21 11:55:41,844 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-21 11:55:41,844 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-21 11:55:41,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:41,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:41,845 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36696 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-21 11:55:41,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:41,845 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1912): 1baade11f622687e2c1237b8441afd49/B is initiating minor compaction (all files) 2023-03-21 11:55:41,846 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1baade11f622687e2c1237b8441afd49/B in TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:41,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:41,846 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/ac41af5f10f0440bba4a366f5df13aeb, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/8446248109cd4bc484adc1e36995b117, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/bb9bdec46c2348a289d415ae05de9ae2] into tmpdir=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp, totalSize=35.8 K 2023-03-21 11:55:41,846 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:41,846 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 48896 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-21 11:55:41,846 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting ac41af5f10f0440bba4a366f5df13aeb, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=454, earliestPutTs=1679399738630 2023-03-21 11:55:41,847 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1912): 1baade11f622687e2c1237b8441afd49/A is initiating minor compaction (all files) 2023-03-21 11:55:41,847 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 1baade11f622687e2c1237b8441afd49/A in TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:41,848 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/081f2d6b58cc4ab59e75485f1a78c819, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/84b1b59a8e8d4ca8b78131c9edca2b4b, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/e5f868adc3ec45018d376cfef4d45689] into tmpdir=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp, totalSize=47.8 K 2023-03-21 11:55:41,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:41,848 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 8446248109cd4bc484adc1e36995b117, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=685, earliestPutTs=1679399739618 2023-03-21 11:55:41,850 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 081f2d6b58cc4ab59e75485f1a78c819, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=454, earliestPutTs=1679399738630 2023-03-21 11:55:41,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:41,850 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 84b1b59a8e8d4ca8b78131c9edca2b4b, keycount=300, bloomtype=ROW, size=19.0 K, encoding=NONE, compression=NONE, seqNum=685, earliestPutTs=1679399739597 2023-03-21 11:55:41,851 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting bb9bdec46c2348a289d415ae05de9ae2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=723, earliestPutTs=1679399739720 2023-03-21 11:55:41,851 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting e5f868adc3ec45018d376cfef4d45689, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=723, earliestPutTs=1679399739709 2023-03-21 11:55:41,852 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:41,852 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:41,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:41,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:41,852 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:41,852 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:55:41,852 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:55:41,852 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:41,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:41,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:41,852 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:55:41,853 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:55:41,853 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:55:41,853 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:41,853 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:41,853 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:41,853 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:41,854 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:41,854 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:41,888 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] throttle.PressureAwareThroughputController(145): 1baade11f622687e2c1237b8441afd49#A#compaction#21 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-21 11:55:41,895 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] throttle.PressureAwareThroughputController(145): 1baade11f622687e2c1237b8441afd49#B#compaction#22 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-21 11:55:41,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(9158): Flush requested on 1baade11f622687e2c1237b8441afd49 2023-03-21 11:55:41,922 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1baade11f622687e2c1237b8441afd49 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-21 11:55:41,923 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1baade11f622687e2c1237b8441afd49, store=A 2023-03-21 11:55:41,923 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:55:41,923 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1baade11f622687e2c1237b8441afd49, store=B 2023-03-21 11:55:41,923 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:55:41,923 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1baade11f622687e2c1237b8441afd49, store=C 2023-03-21 11:55:41,923 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:55:41,933 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/A/dcee3246ab7c4e6293e9f95772fa76bd as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/dcee3246ab7c4e6293e9f95772fa76bd 2023-03-21 11:55:41,953 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1baade11f622687e2c1237b8441afd49/A of 1baade11f622687e2c1237b8441afd49 into dcee3246ab7c4e6293e9f95772fa76bd(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-21 11:55:41,953 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:55:41,953 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1baade11f622687e2c1237b8441afd49: 2023-03-21 11:55:41,958 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49., storeName=1baade11f622687e2c1237b8441afd49/A, priority=13, startTime=1679399741843; duration=0sec 2023-03-21 11:55:41,958 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-21 11:55:41,958 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-21 11:55:41,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:55:41,961 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36696 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-21 11:55:41,961 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1912): 1baade11f622687e2c1237b8441afd49/C is initiating minor compaction (all files) 2023-03-21 11:55:41,961 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 1baade11f622687e2c1237b8441afd49/C in TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:41,962 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/8c5c3e47186c4063b36028ad05769e5d, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/d6252139523742b1ae6cc1cc4414ceca, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/24d0729fddcc4536a16ba5ed97d23127] into tmpdir=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp, totalSize=35.8 K 2023-03-21 11:55:41,962 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=733 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/A/ae55ceef96844e07b1362851cdc0d7a0 2023-03-21 11:55:41,962 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:55:41,964 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 8c5c3e47186c4063b36028ad05769e5d, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=454, earliestPutTs=1679399738630 2023-03-21 11:55:41,966 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting d6252139523742b1ae6cc1cc4414ceca, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=685, earliestPutTs=1679399739618 2023-03-21 11:55:41,967 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 24d0729fddcc4536a16ba5ed97d23127, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=723, earliestPutTs=1679399739720 2023-03-21 11:55:41,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:55:41,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:55:41,973 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:55:41,973 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:41,973 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:55:41,973 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:55:41,974 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:41,974 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:41,975 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:41,975 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:41,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:41,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:41,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:41,976 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:41,976 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:55:41,976 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:41,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:55:41,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:41,977 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:55:41,982 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:55:41,983 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:55:41,983 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:55:41,983 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:41,983 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:55:41,984 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679399741952, max timestamp=1679399741982Num uniques 859; 2023-03-21 11:55:41,984 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-21 11:55:41,985 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679399741952, max timestamp=1679399741982Num uniques 859; 2023-03-21 11:55:41,985 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679399741952, max timestamp=1679399741982Num uniques 859; 2023-03-21 11:55:41,985 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-21 11:55:41,985 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-21 11:55:41,988 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] throttle.PressureAwareThroughputController(145): 1baade11f622687e2c1237b8441afd49#C#compaction#24 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-21 11:55:41,988 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:55:41,988 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:55:41,988 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:41,988 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. started... 2023-03-21 11:55:41,988 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. as already flushing 2023-03-21 11:55:41,989 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:41,989 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:55:41,989 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:55:41,989 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:41,998 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:55:41,998 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:55:41,999 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:41,999 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:42,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:42,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:42,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:42,000 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:42,000 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:55:42,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:42,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:42,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:42,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:42,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:42,012 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:55:42,012 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ab06d58[Count = 0] remaining members to acquire global barrier 2023-03-21 11:55:42,012 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:55:42,012 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:42,016 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:42,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:42,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:42,016 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:55:42,016 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:55:42,016 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:55:42,016 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:42,016 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:55:42,025 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:55:42,025 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:42,025 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:55:42,025 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:42,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:42,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:42,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:42,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:42,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:42,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:42,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:42,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:42,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:42,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:42,033 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:42,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54904 deadline: 1679399802028, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:42,033 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:42,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54864 deadline: 1679399802029, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:42,034 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:42,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:55:42,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54840 deadline: 1679399802029, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:42,034 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:55:42,034 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:55:42,034 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:55:42,034 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:55:42,034 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:55:42,034 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:55:42,034 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:42,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54894 deadline: 1679399802033, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:42,035 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/C/82afd4b6f2a846a9af5f125f12c5fc45 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/82afd4b6f2a846a9af5f125f12c5fc45 2023-03-21 11:55:42,036 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:42,036 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:42,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:42,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:42,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:42,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:42,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:42,038 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:42,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:42,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:42,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:42,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:42,039 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:42,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:42,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:42,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:42,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:42,040 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:42,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:42,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:42,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:42,047 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1baade11f622687e2c1237b8441afd49/C of 1baade11f622687e2c1237b8441afd49 into 82afd4b6f2a846a9af5f125f12c5fc45(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-21 11:55:42,048 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1baade11f622687e2c1237b8441afd49: 2023-03-21 11:55:42,048 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49., storeName=1baade11f622687e2c1237b8441afd49/C, priority=13, startTime=1679399741844; duration=0sec 2023-03-21 11:55:42,048 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:42,048 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:42,048 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-21 11:55:42,048 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:42,048 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:42,048 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:42,048 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:55:42,048 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:55:42,048 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:55:42,048 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:42,048 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:42,048 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:42,048 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:42,048 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:42,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:42,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:42,048 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:55:42,048 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:55:42,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:42,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:42,136 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:42,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54904 deadline: 1679399802134, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:42,137 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:42,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54864 deadline: 1679399802135, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:42,137 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:42,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54840 deadline: 1679399802135, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:42,137 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:42,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54894 deadline: 1679399802135, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:42,149 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:55:42,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:55:42,151 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:55:42,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:55:42,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:55:42,153 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:55:42,153 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:42,153 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:55:42,153 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:55:42,153 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:42,153 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:42,154 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:42,154 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:42,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:42,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:42,154 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:42,154 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:55:42,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:42,155 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:42,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:55:42,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:42,155 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:55:42,155 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:55:42,155 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:55:42,155 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:55:42,155 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:42,155 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:55:42,155 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:55:42,156 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:55:42,156 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:42,156 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. started... 2023-03-21 11:55:42,156 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. as already flushing 2023-03-21 11:55:42,156 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:42,156 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:55:42,156 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:55:42,156 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:42,156 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:55:42,156 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:55:42,157 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:42,157 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:42,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:42,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:42,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:42,157 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:42,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:42,157 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:55:42,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:42,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:42,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:42,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:42,159 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:55:42,159 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7708a574[Count = 0] remaining members to acquire global barrier 2023-03-21 11:55:42,159 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:55:42,159 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:42,159 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:42,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:42,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:42,159 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:55:42,160 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:55:42,160 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:55:42,160 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:42,160 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:55:42,160 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:42,160 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:55:42,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:42,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:42,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:42,160 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:42,161 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:55:42,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:42,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:42,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:42,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:42,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:42,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:42,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:42,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:55:42,163 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:55:42,163 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:55:42,163 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:55:42,163 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:55:42,163 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:55:42,163 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:55:42,164 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:42,164 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:42,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:42,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:42,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:42,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:42,165 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:42,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:42,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:42,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:42,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:42,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:42,166 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:42,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:42,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:42,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:42,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:42,167 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:42,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:42,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:42,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:42,171 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:42,171 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:42,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:42,171 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:55:42,171 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:55:42,171 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:42,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:42,171 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:42,171 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:55:42,171 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:42,171 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:55:42,171 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:55:42,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:42,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:42,172 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:42,172 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:42,172 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:42,172 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:42,172 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:42,272 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:55:42,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:55:42,279 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:55:42,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:55:42,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:55:42,290 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:55:42,290 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:42,290 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:55:42,290 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:55:42,290 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:42,290 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:42,291 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:42,291 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:42,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:42,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:42,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:42,292 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:42,292 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:55:42,292 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:42,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:55:42,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:42,293 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:55:42,301 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:55:42,301 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:55:42,301 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:55:42,301 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:42,301 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:55:42,306 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:55:42,306 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:42,306 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. started... 2023-03-21 11:55:42,306 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:55:42,306 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. as already flushing 2023-03-21 11:55:42,306 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:42,306 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:55:42,307 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:55:42,307 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:42,307 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:55:42,307 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:55:42,308 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:42,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:42,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:42,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:42,308 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:42,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:42,309 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:42,309 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:55:42,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:42,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:42,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:42,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:42,311 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:55:42,311 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@335f0b18[Count = 0] remaining members to acquire global barrier 2023-03-21 11:55:42,311 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:55:42,311 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:42,314 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:42,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:42,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:42,315 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:55:42,315 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:55:42,315 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:55:42,316 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:42,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:42,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:42,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:42,316 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:55:42,316 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:42,316 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:55:42,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:42,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:42,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:42,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:42,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:42,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:42,318 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:42,318 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:55:42,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:42,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:55:42,320 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:55:42,320 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:55:42,320 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:55:42,320 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:55:42,320 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:55:42,320 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:55:42,321 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:42,321 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:42,321 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:42,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:42,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:42,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:42,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:42,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:42,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:42,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:42,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:42,324 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:42,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:42,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:42,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:42,334 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:42,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:42,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:42,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:42,339 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:42,339 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:42,339 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:42,339 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:42,339 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:42,340 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:42,340 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:42,339 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:42,340 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:42,340 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/B/a9e65d306c2d439d8adece1adacca899 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/a9e65d306c2d439d8adece1adacca899 2023-03-21 11:55:42,340 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:42,340 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:55:42,341 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:55:42,341 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:55:42,341 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:55:42,342 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:42,343 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:55:42,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54904 deadline: 1679399802338, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:42,343 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:42,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54894 deadline: 1679399802339, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:42,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:42,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:42,344 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:42,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54864 deadline: 1679399802340, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:42,345 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:42,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54840 deadline: 1679399802340, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:42,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:42,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:42,356 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1baade11f622687e2c1237b8441afd49/B of 1baade11f622687e2c1237b8441afd49 into a9e65d306c2d439d8adece1adacca899(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-21 11:55:42,357 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1baade11f622687e2c1237b8441afd49: 2023-03-21 11:55:42,357 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49., storeName=1baade11f622687e2c1237b8441afd49/B, priority=13, startTime=1679399741844; duration=0sec 2023-03-21 11:55:42,357 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-21 11:55:42,424 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=733 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/B/e6fbe49c71ea4a30b61e71867fb7a879 2023-03-21 11:55:42,442 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:55:42,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:55:42,446 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=733 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/C/dbca2fc359834f2dbe37ff32cf1578eb 2023-03-21 11:55:42,447 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:55:42,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:55:42,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:55:42,450 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:55:42,451 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:42,451 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:55:42,451 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:55:42,452 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:42,452 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:42,453 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:42,453 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:42,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:42,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:42,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:42,453 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:42,453 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:55:42,454 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:42,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:55:42,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:42,455 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/A/ae55ceef96844e07b1362851cdc0d7a0 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/ae55ceef96844e07b1362851cdc0d7a0 2023-03-21 11:55:42,455 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:55:42,455 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:55:42,455 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:55:42,455 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:55:42,455 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:42,455 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:55:42,456 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:55:42,456 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:55:42,456 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:42,456 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. started... 2023-03-21 11:55:42,456 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. as already flushing 2023-03-21 11:55:42,456 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:42,456 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:55:42,456 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:55:42,456 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:42,456 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:55:42,456 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:55:42,458 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:42,458 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:42,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:42,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:42,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:42,458 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:42,458 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:55:42,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:42,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:42,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:42,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:42,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:42,460 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:55:42,460 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@68bce5d8[Count = 0] remaining members to acquire global barrier 2023-03-21 11:55:42,460 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:55:42,460 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:42,466 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/ae55ceef96844e07b1362851cdc0d7a0, entries=150, sequenceid=733, filesize=11.9 K 2023-03-21 11:55:42,468 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/B/e6fbe49c71ea4a30b61e71867fb7a879 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/e6fbe49c71ea4a30b61e71867fb7a879 2023-03-21 11:55:42,476 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/e6fbe49c71ea4a30b61e71867fb7a879, entries=150, sequenceid=733, filesize=11.9 K 2023-03-21 11:55:42,480 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/C/dbca2fc359834f2dbe37ff32cf1578eb as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/dbca2fc359834f2dbe37ff32cf1578eb 2023-03-21 11:55:42,481 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:42,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:42,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:42,482 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:55:42,482 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:55:42,482 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:55:42,483 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:42,484 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:55:42,484 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:42,484 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:55:42,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:42,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:42,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:42,484 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:42,485 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:55:42,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:42,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:42,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:42,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:42,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:42,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:42,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:42,489 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/dbca2fc359834f2dbe37ff32cf1578eb, entries=150, sequenceid=733, filesize=11.9 K 2023-03-21 11:55:42,491 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 1baade11f622687e2c1237b8441afd49 in 568ms, sequenceid=733, compaction requested=false 2023-03-21 11:55:42,491 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1baade11f622687e2c1237b8441afd49: 2023-03-21 11:55:42,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:55:42,491 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:55:42,491 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:55:42,491 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:55:42,491 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:55:42,491 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:55:42,491 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:55:42,498 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:42,498 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:42,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:42,498 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:42,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:42,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:42,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:42,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:42,498 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:42,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:42,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:42,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:42,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:42,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:42,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:42,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:42,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:42,504 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:42,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:42,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:42,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:42,506 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:42,506 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:42,506 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:42,506 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:42,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:42,506 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:55:42,507 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:55:42,506 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:42,506 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:42,507 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:55:42,507 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:42,507 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:42,507 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:42,507 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:42,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:42,507 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:55:42,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:42,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:42,607 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:55:42,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:55:42,610 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:55:42,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:55:42,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:55:42,623 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:55:42,623 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:42,623 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:55:42,623 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:55:42,624 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:42,624 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:42,625 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:42,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:42,625 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:42,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:42,625 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:42,625 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:55:42,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:42,626 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:42,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:55:42,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:42,626 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:55:42,626 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:55:42,627 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:55:42,627 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:55:42,627 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:42,627 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:55:42,630 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:55:42,630 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:42,630 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:55:42,630 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. started... 2023-03-21 11:55:42,630 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1baade11f622687e2c1237b8441afd49 3/3 column families, dataSize=134.18 KB heapSize=375.61 KB 2023-03-21 11:55:42,631 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1baade11f622687e2c1237b8441afd49, store=A 2023-03-21 11:55:42,631 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-21 11:55:42,631 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1baade11f622687e2c1237b8441afd49, store=B 2023-03-21 11:55:42,631 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-21 11:55:42,631 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1baade11f622687e2c1237b8441afd49, store=C 2023-03-21 11:55:42,631 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-21 11:55:42,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(9158): Flush requested on 1baade11f622687e2c1237b8441afd49 2023-03-21 11:55:42,651 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. as already flushing 2023-03-21 11:55:42,659 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=771 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/A/6eb0e3139e9743119d8eb5541ba69c5c 2023-03-21 11:55:42,665 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:42,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54894 deadline: 1679399802662, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:42,666 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:42,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54904 deadline: 1679399802663, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:42,666 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:42,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54864 deadline: 1679399802663, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:42,670 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:42,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54840 deadline: 1679399802670, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:42,768 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:42,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54894 deadline: 1679399802767, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:42,768 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:42,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54864 deadline: 1679399802767, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:42,773 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:42,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54840 deadline: 1679399802771, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:42,774 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:42,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54904 deadline: 1679399802771, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:42,969 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:42,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54894 deadline: 1679399802969, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:42,972 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:42,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54864 deadline: 1679399802970, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:42,976 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:42,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54840 deadline: 1679399802975, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:42,977 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:42,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54904 deadline: 1679399802975, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:43,091 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=771 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/B/d93b8a06415240c99c27a1613c713c3f 2023-03-21 11:55:43,275 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:43,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54894 deadline: 1679399803273, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:43,276 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:43,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54864 deadline: 1679399803276, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:43,281 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:43,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54904 deadline: 1679399803278, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:43,281 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:43,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54840 deadline: 1679399803280, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:43,518 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=771 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/C/b76e30e438d0471e999a4b13dde5d7f6 2023-03-21 11:55:43,524 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/A/6eb0e3139e9743119d8eb5541ba69c5c as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/6eb0e3139e9743119d8eb5541ba69c5c 2023-03-21 11:55:43,531 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/6eb0e3139e9743119d8eb5541ba69c5c, entries=150, sequenceid=771, filesize=11.9 K 2023-03-21 11:55:43,533 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/B/d93b8a06415240c99c27a1613c713c3f as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/d93b8a06415240c99c27a1613c713c3f 2023-03-21 11:55:43,540 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/d93b8a06415240c99c27a1613c713c3f, entries=150, sequenceid=771, filesize=11.9 K 2023-03-21 11:55:43,542 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/C/b76e30e438d0471e999a4b13dde5d7f6 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/b76e30e438d0471e999a4b13dde5d7f6 2023-03-21 11:55:43,548 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/b76e30e438d0471e999a4b13dde5d7f6, entries=150, sequenceid=771, filesize=11.9 K 2023-03-21 11:55:43,549 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~375.56 KB/384576, currentSize=46.96 KB/48090 for 1baade11f622687e2c1237b8441afd49 in 919ms, sequenceid=771, compaction requested=true 2023-03-21 11:55:43,550 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1baade11f622687e2c1237b8441afd49: 2023-03-21 11:55:43,550 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:43,550 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:55:43,551 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:55:43,551 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:43,551 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:55:43,551 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:55:43,552 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:43,552 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:43,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:43,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:43,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:43,552 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:43,552 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:55:43,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:43,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:43,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:43,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:43,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:43,553 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:55:43,553 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:55:43,553 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@61ef9f3a[Count = 0] remaining members to acquire global barrier 2023-03-21 11:55:43,553 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:43,554 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:43,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:43,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:43,554 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:55:43,554 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:55:43,554 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:55:43,554 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:43,554 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:55:43,555 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:43,555 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:55:43,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:43,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:43,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:43,555 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:43,555 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:55:43,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:43,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:43,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:43,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:43,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:43,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:43,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:43,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:55:43,557 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:55:43,557 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:55:43,557 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:55:43,557 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:55:43,557 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:55:43,557 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:55:43,559 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:43,559 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:43,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:43,559 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:43,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:43,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:43,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:43,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:43,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:43,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:43,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:43,560 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:43,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:43,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:43,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:43,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:43,561 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:43,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:43,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:43,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:43,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:43,563 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:43,563 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:43,563 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:43,563 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:43,563 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:43,563 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:55:43,563 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:43,563 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:55:43,563 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:55:43,563 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:43,563 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:43,563 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:43,563 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:43,563 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:55:43,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:43,563 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:55:43,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:43,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:43,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:43,663 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:55:43,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:55:43,666 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:55:43,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:55:43,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:55:43,667 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:55:43,667 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:43,667 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:55:43,667 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:55:43,668 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:43,668 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:43,668 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:43,668 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:43,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:43,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:43,669 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:43,669 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:55:43,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:43,669 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:43,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:55:43,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:43,669 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:55:43,670 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:55:43,670 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:55:43,670 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:55:43,670 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:43,670 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:55:43,670 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:55:43,670 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:55:43,670 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:43,670 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. started... 2023-03-21 11:55:43,670 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1baade11f622687e2c1237b8441afd49 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-21 11:55:43,670 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1baade11f622687e2c1237b8441afd49, store=A 2023-03-21 11:55:43,671 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:55:43,671 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1baade11f622687e2c1237b8441afd49, store=B 2023-03-21 11:55:43,671 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:55:43,671 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1baade11f622687e2c1237b8441afd49, store=C 2023-03-21 11:55:43,671 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:55:43,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(9158): Flush requested on 1baade11f622687e2c1237b8441afd49 2023-03-21 11:55:43,780 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. as already flushing 2023-03-21 11:55:43,798 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679399743793, max timestamp=1679399743797Num uniques 859; 2023-03-21 11:55:43,799 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679399743793, max timestamp=1679399743797Num uniques 859; 2023-03-21 11:55:43,799 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679399743793, max timestamp=1679399743797Num uniques 859; 2023-03-21 11:55:43,799 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-21 11:55:43,799 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-21 11:55:43,799 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-21 11:55:43,817 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:43,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54892 deadline: 1679399803812, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:43,821 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:43,821 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:43,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54840 deadline: 1679399803816, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:43,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54864 deadline: 1679399803816, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:43,821 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:43,821 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:43,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54904 deadline: 1679399803817, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:43,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54894 deadline: 1679399803817, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:43,920 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:43,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54892 deadline: 1679399803919, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:43,924 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:43,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54864 deadline: 1679399803923, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:43,924 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:43,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54840 deadline: 1679399803924, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:43,928 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:43,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54904 deadline: 1679399803924, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:43,930 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:43,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54894 deadline: 1679399803925, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:44,094 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=781 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/A/7520b7ea1bbf4a8d8b901fb9700d9d48 2023-03-21 11:55:44,115 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=781 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/B/abcc8dbaa4504506ac0769fec5df6e44 2023-03-21 11:55:44,126 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:44,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54864 deadline: 1679399804126, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:44,129 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:44,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54840 deadline: 1679399804127, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:44,130 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:44,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54892 deadline: 1679399804127, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:44,134 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:44,135 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:44,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54904 deadline: 1679399804131, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:44,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54894 deadline: 1679399804131, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:44,434 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:44,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54864 deadline: 1679399804429, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:44,436 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:44,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54840 deadline: 1679399804432, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:44,436 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:44,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54892 deadline: 1679399804433, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:44,438 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:44,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54904 deadline: 1679399804438, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:44,439 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:44,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54894 deadline: 1679399804438, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:44,539 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=781 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/C/5ff8d939f3494163bf8bbe3f10e94e84 2023-03-21 11:55:44,546 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/A/7520b7ea1bbf4a8d8b901fb9700d9d48 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/7520b7ea1bbf4a8d8b901fb9700d9d48 2023-03-21 11:55:44,553 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/7520b7ea1bbf4a8d8b901fb9700d9d48, entries=150, sequenceid=781, filesize=11.9 K 2023-03-21 11:55:44,555 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/B/abcc8dbaa4504506ac0769fec5df6e44 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/abcc8dbaa4504506ac0769fec5df6e44 2023-03-21 11:55:44,561 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/abcc8dbaa4504506ac0769fec5df6e44, entries=150, sequenceid=781, filesize=11.9 K 2023-03-21 11:55:44,562 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/C/5ff8d939f3494163bf8bbe3f10e94e84 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/5ff8d939f3494163bf8bbe3f10e94e84 2023-03-21 11:55:44,570 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/5ff8d939f3494163bf8bbe3f10e94e84, entries=150, sequenceid=781, filesize=11.9 K 2023-03-21 11:55:44,572 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 1baade11f622687e2c1237b8441afd49 in 901ms, sequenceid=781, compaction requested=true 2023-03-21 11:55:44,572 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1baade11f622687e2c1237b8441afd49: 2023-03-21 11:55:44,572 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:44,573 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:55:44,573 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:55:44,573 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:44,573 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:55:44,573 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:55:44,574 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:44,574 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:44,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:44,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:44,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:44,575 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:44,575 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:55:44,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:44,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:44,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:44,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:44,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:44,576 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:55:44,576 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@537ea576[Count = 0] remaining members to acquire global barrier 2023-03-21 11:55:44,576 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:55:44,576 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:44,577 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:44,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:44,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:44,577 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:55:44,577 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:44,577 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:55:44,577 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:55:44,577 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:55:44,578 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:44,578 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:55:44,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:44,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:44,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:44,578 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:44,578 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:55:44,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:44,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:44,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:44,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:44,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:44,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:44,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:44,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:55:44,580 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:55:44,580 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:55:44,580 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:55:44,580 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:55:44,580 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:55:44,580 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:55:44,581 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:44,581 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:44,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:44,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:44,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:44,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:44,582 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:44,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:44,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:44,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:44,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:44,582 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:44,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:44,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:44,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:44,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:44,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:44,583 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:44,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:44,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:44,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:44,586 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:44,586 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:44,586 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:44,586 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:44,586 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:44,586 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:55:44,586 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:55:44,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:44,586 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:55:44,586 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:44,586 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:44,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:44,587 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:44,587 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:44,587 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:55:44,587 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:44,587 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:55:44,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:44,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:44,687 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:55:44,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:55:44,689 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:55:44,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:55:44,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:55:44,690 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:55:44,690 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:44,690 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:55:44,690 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:55:44,691 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:44,691 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:44,692 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:44,692 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:44,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:44,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:44,693 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:44,693 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:55:44,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:44,693 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:44,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:55:44,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:44,693 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:55:44,694 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:55:44,694 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:55:44,694 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:55:44,694 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:44,694 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:55:44,695 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:55:44,695 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:55:44,695 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:44,695 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. started... 2023-03-21 11:55:44,695 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1baade11f622687e2c1237b8441afd49 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-03-21 11:55:44,695 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1baade11f622687e2c1237b8441afd49, store=A 2023-03-21 11:55:44,695 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-21 11:55:44,695 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1baade11f622687e2c1237b8441afd49, store=B 2023-03-21 11:55:44,695 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-21 11:55:44,695 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1baade11f622687e2c1237b8441afd49, store=C 2023-03-21 11:55:44,695 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-21 11:55:44,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(9158): Flush requested on 1baade11f622687e2c1237b8441afd49 2023-03-21 11:55:44,939 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. as already flushing 2023-03-21 11:55:44,949 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:44,949 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:44,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54840 deadline: 1679399804946, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:44,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54904 deadline: 1679399804946, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:44,949 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:44,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54894 deadline: 1679399804947, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:44,950 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:44,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54892 deadline: 1679399804948, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:44,952 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:44,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54864 deadline: 1679399804949, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:45,055 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:45,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54894 deadline: 1679399805051, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:45,056 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:45,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54864 deadline: 1679399805055, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:45,056 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:45,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54892 deadline: 1679399805055, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:45,104 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=817 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/A/3e450e6d72af4f6f8a153fea77e3623c 2023-03-21 11:55:45,125 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=817 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/B/83de8f130b27469f95deb08d73b65e0e 2023-03-21 11:55:45,155 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=817 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/C/45d08817049145c087900115258424d4 2023-03-21 11:55:45,163 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/A/3e450e6d72af4f6f8a153fea77e3623c as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/3e450e6d72af4f6f8a153fea77e3623c 2023-03-21 11:55:45,172 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/3e450e6d72af4f6f8a153fea77e3623c, entries=150, sequenceid=817, filesize=11.9 K 2023-03-21 11:55:45,177 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/B/83de8f130b27469f95deb08d73b65e0e as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/83de8f130b27469f95deb08d73b65e0e 2023-03-21 11:55:45,185 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/83de8f130b27469f95deb08d73b65e0e, entries=150, sequenceid=817, filesize=11.9 K 2023-03-21 11:55:45,187 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/C/45d08817049145c087900115258424d4 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/45d08817049145c087900115258424d4 2023-03-21 11:55:45,197 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/45d08817049145c087900115258424d4, entries=150, sequenceid=817, filesize=11.9 K 2023-03-21 11:55:45,198 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for 1baade11f622687e2c1237b8441afd49 in 503ms, sequenceid=817, compaction requested=true 2023-03-21 11:55:45,198 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1baade11f622687e2c1237b8441afd49: 2023-03-21 11:55:45,198 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:45,198 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:55:45,199 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:55:45,199 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:45,199 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:55:45,199 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:55:45,201 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:45,201 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:45,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:45,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:45,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:45,201 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:45,201 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:55:45,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:45,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:45,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:45,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:45,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:45,203 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:55:45,203 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@183263ca[Count = 0] remaining members to acquire global barrier 2023-03-21 11:55:45,203 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:55:45,203 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:45,204 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:45,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:45,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:45,204 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:55:45,204 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:55:45,204 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:55:45,204 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:45,204 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:55:45,205 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:55:45,205 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:45,205 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:45,206 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:55:45,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:45,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:45,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:45,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:45,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:45,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:45,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:45,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:45,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:45,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:45,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:55:45,209 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:55:45,209 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:55:45,209 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:55:45,209 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:55:45,209 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:55:45,209 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:55:45,223 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:45,223 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:45,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:45,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:45,223 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:45,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:45,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:45,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:45,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:45,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:45,223 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:45,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:45,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:45,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:45,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:45,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:45,225 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:45,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:45,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:45,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:45,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:45,231 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:45,231 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:45,231 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:45,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:45,231 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:45,231 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:45,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:45,232 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:45,232 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:45,232 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:45,232 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:45,232 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:55:45,232 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:55:45,232 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:45,232 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:55:45,232 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:45,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:45,232 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:55:45,232 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:55:45,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(9158): Flush requested on 1baade11f622687e2c1237b8441afd49 2023-03-21 11:55:45,261 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1baade11f622687e2c1237b8441afd49 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-21 11:55:45,262 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1baade11f622687e2c1237b8441afd49, store=A 2023-03-21 11:55:45,263 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:55:45,263 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1baade11f622687e2c1237b8441afd49, store=B 2023-03-21 11:55:45,263 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:55:45,263 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1baade11f622687e2c1237b8441afd49, store=C 2023-03-21 11:55:45,263 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:55:45,300 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:45,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54892 deadline: 1679399805297, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:45,303 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:45,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54864 deadline: 1679399805300, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:45,303 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:45,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54894 deadline: 1679399805300, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:45,333 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:55:45,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:55:45,338 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:55:45,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:55:45,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:55:45,346 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:55:45,346 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:45,346 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:55:45,346 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:55:45,354 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:45,354 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:45,355 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:45,355 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:45,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:45,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:45,356 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:45,356 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:55:45,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:45,357 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:45,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:55:45,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:45,358 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:55:45,359 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:55:45,359 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:55:45,359 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:55:45,359 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:45,359 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:55:45,359 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:55:45,359 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:55:45,359 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:45,359 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. started... 2023-03-21 11:55:45,359 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. as already flushing 2023-03-21 11:55:45,359 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:45,359 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:55:45,359 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:55:45,359 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:45,359 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:55:45,360 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:55:45,361 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:45,361 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:45,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:45,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:45,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:45,361 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:45,362 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:55:45,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:45,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:45,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:45,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:45,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:45,363 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:55:45,363 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@172fd731[Count = 0] remaining members to acquire global barrier 2023-03-21 11:55:45,363 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:55:45,364 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:45,364 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:45,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:45,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:45,365 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:45,365 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:55:45,365 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:55:45,365 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:55:45,365 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:55:45,366 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:55:45,366 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:45,366 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:55:45,366 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:45,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:45,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:45,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:45,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:45,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:45,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:45,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:45,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:45,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:45,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:45,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:55:45,369 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:55:45,369 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:55:45,369 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:55:45,369 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:55:45,369 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:55:45,369 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:55:45,371 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:45,371 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:45,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:45,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:45,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:45,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:45,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:45,372 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:45,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:45,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:45,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:45,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:45,375 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:45,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:45,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:45,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:45,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:45,377 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:45,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:45,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:45,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:45,381 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:45,381 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:45,381 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:45,381 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:55:45,381 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:55:45,381 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:45,381 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:45,381 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:45,381 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:45,382 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:45,382 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:45,382 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:45,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:45,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:45,382 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:55:45,382 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:55:45,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:45,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:45,413 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:45,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54864 deadline: 1679399805406, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:45,413 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:45,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54894 deadline: 1679399805407, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:45,419 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:45,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54892 deadline: 1679399805402, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:45,482 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:55:45,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:55:45,484 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:55:45,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:55:45,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:55:45,487 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:55:45,487 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:45,487 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:55:45,487 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:55:45,488 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:45,488 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:45,489 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:45,489 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:45,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:45,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:45,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:45,489 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:45,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:55:45,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:45,490 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:55:45,490 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:55:45,490 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:55:45,490 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:55:45,490 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:45,490 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:55:45,490 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:55:45,490 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:55:45,490 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:45,490 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:45,490 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:55:45,491 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. started... 2023-03-21 11:55:45,491 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. as already flushing 2023-03-21 11:55:45,491 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:45,491 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:55:45,491 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:55:45,491 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:45,491 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:55:45,491 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:55:45,492 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:45,492 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:45,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:45,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:45,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:45,493 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:45,493 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:55:45,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:45,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:45,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:45,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:45,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:45,494 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:55:45,494 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@aa49ca1[Count = 0] remaining members to acquire global barrier 2023-03-21 11:55:45,494 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:55:45,494 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:45,495 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:45,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:45,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:45,495 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:55:45,495 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:55:45,495 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:55:45,495 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:45,495 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:55:45,496 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:55:45,496 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:45,496 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:55:45,496 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:45,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:45,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:45,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:45,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:45,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:45,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:45,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:45,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:45,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:45,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:45,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:55:45,499 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:55:45,499 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:55:45,499 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:55:45,499 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:55:45,499 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:55:45,499 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:55:45,500 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:45,500 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:45,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:45,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:45,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:45,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:45,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:45,500 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:45,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:45,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:45,502 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:45,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:45,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:45,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:45,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:45,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:45,504 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:45,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:45,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:45,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:45,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:45,513 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:45,513 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:45,513 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:45,513 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:45,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:45,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:45,513 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:45,513 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:55:45,513 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:55:45,513 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:45,513 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:55:45,513 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:55:45,513 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:45,513 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:55:45,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:45,514 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:45,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:45,514 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:45,514 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:45,614 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:55:45,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:55:45,616 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:55:45,616 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:45,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54894 deadline: 1679399805616, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:45,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:55:45,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:55:45,618 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:55:45,618 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:45,618 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:55:45,618 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:55:45,618 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:45,618 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:45,619 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:45,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54864 deadline: 1679399805617, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:45,619 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:45,619 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:45,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:45,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:45,620 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:45,620 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:55:45,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:45,620 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:45,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:55:45,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:45,620 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:55:45,626 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:55:45,626 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:55:45,626 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:55:45,626 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:45,626 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:55:45,630 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:55:45,630 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:55:45,630 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:45,630 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. started... 2023-03-21 11:55:45,630 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. as already flushing 2023-03-21 11:55:45,630 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:45,630 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:55:45,630 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:55:45,630 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:45,630 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:55:45,630 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:55:45,632 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:45,632 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:45,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:45,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:45,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:45,633 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:45,633 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:55:45,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:45,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:45,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:45,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:45,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:45,634 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:45,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54892 deadline: 1679399805632, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:45,636 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:55:45,636 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c57dd96[Count = 0] remaining members to acquire global barrier 2023-03-21 11:55:45,636 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:55:45,636 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:45,637 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:45,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:45,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:45,637 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:45,637 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:55:45,637 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:55:45,637 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:55:45,637 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:55:45,638 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:45,638 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:55:45,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:45,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:45,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:45,638 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:45,638 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:55:45,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:45,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:45,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:45,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:45,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:45,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:45,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:45,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:55:45,640 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:55:45,640 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:55:45,640 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:55:45,640 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:55:45,640 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:55:45,640 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:55:45,642 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:45,642 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:45,642 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:45,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:45,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:45,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:45,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:45,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:45,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:45,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:45,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:45,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:45,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:45,643 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:45,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:45,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:45,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:45,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:45,644 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:45,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:45,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:45,646 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:45,646 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:45,646 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:45,646 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:45,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:45,646 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:45,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:45,646 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:45,646 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:45,646 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:45,646 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:45,646 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:45,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:45,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:45,650 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:55:45,650 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:55:45,650 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:55:45,650 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:55:45,699 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=830 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/A/4fbc9568fd394997823c8df520ccf30f 2023-03-21 11:55:45,750 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:55:45,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:55:45,753 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:55:45,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:55:45,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:55:45,755 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:55:45,755 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:45,755 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:55:45,755 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:55:45,755 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:45,755 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:45,756 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:45,756 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:45,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:45,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:45,756 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:45,756 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:55:45,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:45,757 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:45,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:55:45,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:45,757 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:55:45,757 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:55:45,757 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:55:45,757 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:55:45,757 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:45,757 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:55:45,758 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:55:45,758 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:55:45,758 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:45,758 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. started... 2023-03-21 11:55:45,758 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. as already flushing 2023-03-21 11:55:45,758 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:45,758 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:55:45,758 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:55:45,758 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:45,758 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:55:45,758 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:55:45,759 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:45,759 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:45,760 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:45,760 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:55:45,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:45,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:45,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:45,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:45,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:45,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:45,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:45,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:45,761 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:55:45,761 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:55:45,761 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@27f0d7e5[Count = 0] remaining members to acquire global barrier 2023-03-21 11:55:45,761 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:45,762 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:45,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:45,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:45,762 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:55:45,762 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:55:45,762 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:55:45,763 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:45,763 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:55:45,763 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:45,764 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:55:45,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:45,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:45,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:45,764 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:45,764 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:55:45,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:45,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:45,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:45,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:45,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:45,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:45,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:45,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:55:45,766 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:55:45,766 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:55:45,766 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:55:45,766 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:55:45,766 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:55:45,766 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:55:45,768 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:45,768 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:45,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:45,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:45,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:45,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:45,768 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:45,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:45,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:45,768 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:45,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:45,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:45,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:45,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:45,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:45,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:45,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:45,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:45,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:45,770 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:45,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:45,772 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:45,772 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:45,772 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:45,772 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:45,772 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:45,772 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:55:45,772 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:55:45,772 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:45,772 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:45,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:45,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:45,772 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:55:45,772 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:55:45,772 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:45,772 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:45,773 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:45,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:45,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:45,873 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:55:45,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:55:45,877 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:55:45,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:55:45,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:55:45,878 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:55:45,878 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:45,878 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:55:45,878 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:55:45,878 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:45,879 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:45,879 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:45,879 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:45,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:45,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:45,880 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:45,880 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:55:45,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:45,880 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:45,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:55:45,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:45,880 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:55:45,881 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:55:45,881 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:55:45,881 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:55:45,881 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:45,881 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:55:45,881 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:55:45,881 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:55:45,881 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:45,881 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. started... 2023-03-21 11:55:45,881 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. as already flushing 2023-03-21 11:55:45,881 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:45,881 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:55:45,882 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:55:45,882 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:45,882 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:55:45,882 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:55:45,883 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:45,883 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:45,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:45,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:45,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:45,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:45,883 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:45,883 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:55:45,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:45,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:45,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:45,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:45,885 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:55:45,885 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4e8f1a75[Count = 0] remaining members to acquire global barrier 2023-03-21 11:55:45,885 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:55:45,885 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:45,886 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:45,886 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:55:45,888 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:45,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:45,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:45,888 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:55:45,888 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:55:45,888 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:55:45,889 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:45,889 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:55:45,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:45,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:45,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:45,889 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:45,890 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:55:45,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:45,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:45,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:45,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:45,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:45,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:45,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:45,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:55:45,892 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:55:45,892 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:55:45,892 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:55:45,892 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:55:45,892 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:55:45,892 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:55:45,893 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:45,893 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:45,893 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:45,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:45,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:45,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:45,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:45,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:45,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:45,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:45,894 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:45,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:45,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:45,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:45,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:45,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:45,895 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:45,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:45,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:45,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:45,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:45,897 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:45,897 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:45,897 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:45,897 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:45,897 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:45,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:45,897 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:45,897 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:55:45,897 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:55:45,897 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:45,897 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:45,897 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:45,898 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:45,898 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:55:45,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:45,898 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:55:45,898 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:55:45,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:45,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:45,920 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:45,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54894 deadline: 1679399805920, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:45,923 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:45,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54864 deadline: 1679399805921, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:45,942 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:45,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54892 deadline: 1679399805938, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:45,960 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:45,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54840 deadline: 1679399805958, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:45,960 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:45,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54904 deadline: 1679399805959, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:45,998 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:55:45,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:55:46,000 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:55:46,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:55:46,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:55:46,001 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:55:46,001 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:46,001 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:55:46,001 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:55:46,002 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:46,002 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:46,002 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:46,003 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:46,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:46,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:46,003 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:46,003 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:55:46,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:46,003 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:46,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:55:46,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:46,004 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:55:46,010 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:55:46,010 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:55:46,010 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:55:46,010 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:46,010 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:55:46,011 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:55:46,011 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:55:46,011 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:46,011 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. started... 2023-03-21 11:55:46,011 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. as already flushing 2023-03-21 11:55:46,011 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:46,011 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:55:46,011 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:55:46,011 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:46,011 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:55:46,011 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:55:46,014 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:46,014 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:46,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:46,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:46,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:46,014 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:46,015 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:55:46,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:46,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:46,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:46,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:46,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:46,016 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:55:46,016 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@60e0d691[Count = 0] remaining members to acquire global barrier 2023-03-21 11:55:46,016 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:55:46,016 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:46,022 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:46,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:46,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:46,023 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:46,023 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:55:46,023 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:55:46,023 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:55:46,023 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:55:46,024 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:46,024 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:55:46,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:46,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:46,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:46,024 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:46,025 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:55:46,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:46,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:46,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:46,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:46,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:46,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:46,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:46,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:55:46,026 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:55:46,026 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:55:46,026 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:55:46,026 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:55:46,026 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:55:46,027 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:55:46,027 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:46,027 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:46,028 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:46,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:46,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:46,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:46,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:46,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:46,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:46,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:46,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:46,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:46,030 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:46,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:46,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:46,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:46,031 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:46,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:46,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:46,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:46,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:46,040 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:46,040 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:46,040 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:46,040 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:46,040 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:46,040 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:46,040 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:46,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:46,040 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:46,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:46,040 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:46,040 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:55:46,040 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:55:46,040 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:46,040 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:55:46,040 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:55:46,040 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:55:46,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:46,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:46,122 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=830 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/B/ae65b0e384ad4358a63dc9ab53d5e9f7 2023-03-21 11:55:46,141 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:55:46,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:55:46,143 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:55:46,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:55:46,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:55:46,143 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:55:46,144 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:46,144 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:55:46,144 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:55:46,144 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:46,144 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:46,145 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:46,145 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:46,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:46,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:46,145 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:46,145 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:55:46,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:46,145 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:46,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:55:46,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:46,146 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:55:46,146 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:55:46,146 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:55:46,146 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:55:46,146 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:46,146 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:55:46,146 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:55:46,146 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:55:46,146 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:46,146 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. started... 2023-03-21 11:55:46,146 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. as already flushing 2023-03-21 11:55:46,146 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:46,146 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:55:46,146 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:55:46,146 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:46,146 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:55:46,146 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:55:46,147 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:46,147 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:46,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:46,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:46,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:46,147 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:46,147 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:55:46,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:46,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:46,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:46,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:46,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:46,148 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:55:46,149 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@72491c76[Count = 0] remaining members to acquire global barrier 2023-03-21 11:55:46,149 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:55:46,149 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:46,154 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:46,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:46,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:46,154 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:46,154 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:55:46,154 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:55:46,154 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:55:46,154 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:55:46,155 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:46,155 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:55:46,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:46,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:46,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:46,155 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:46,155 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:55:46,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:46,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:46,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:46,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:46,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:46,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:46,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:46,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:55:46,157 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:55:46,157 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:55:46,157 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:55:46,157 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:55:46,157 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:55:46,157 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:55:46,160 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:46,160 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:46,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:46,160 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:46,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:46,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:46,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:46,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:46,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:46,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:46,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:46,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:46,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:46,170 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:46,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:46,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:46,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:46,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:46,172 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:46,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:46,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:46,174 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:46,174 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:46,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:46,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:46,174 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:46,174 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:46,175 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:46,175 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:46,175 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:46,175 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:46,175 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:46,175 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:55:46,175 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:55:46,175 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:46,175 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:55:46,175 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:55:46,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:46,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:46,275 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:55:46,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:55:46,277 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:55:46,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:55:46,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:55:46,278 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:55:46,278 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:46,278 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:55:46,278 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:55:46,279 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:46,279 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:46,280 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:46,280 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:46,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:46,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:46,280 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:46,280 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:55:46,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:46,280 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:46,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:55:46,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:46,280 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:55:46,281 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:55:46,281 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:55:46,281 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:55:46,281 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:46,281 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:55:46,281 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:55:46,281 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:55:46,281 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:46,281 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. started... 2023-03-21 11:55:46,281 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. as already flushing 2023-03-21 11:55:46,281 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:46,281 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:55:46,281 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:55:46,281 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:46,281 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:55:46,281 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:55:46,282 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:46,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:46,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:46,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:46,282 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:46,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:46,283 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:46,283 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:55:46,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:46,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:46,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:46,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:46,284 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:55:46,284 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@53a7b225[Count = 0] remaining members to acquire global barrier 2023-03-21 11:55:46,284 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:55:46,284 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:46,284 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:46,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:46,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:46,284 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:55:46,284 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:55:46,284 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:46,284 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:55:46,284 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:55:46,285 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:46,285 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:55:46,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:46,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:46,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:46,285 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:46,285 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:55:46,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:46,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:46,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:46,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:46,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:46,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:46,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:46,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:55:46,287 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:55:46,287 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:55:46,287 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:55:46,287 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:55:46,287 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:55:46,287 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:55:46,290 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:46,290 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:46,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:46,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:46,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:46,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:46,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:46,290 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:46,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:46,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:46,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:46,290 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:46,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:46,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:46,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:46,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:46,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:46,292 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:46,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:46,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:46,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:46,293 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:46,293 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:46,293 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:46,293 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:46,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:46,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:46,293 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:46,293 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:55:46,293 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:55:46,293 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:46,294 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:46,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:46,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:46,294 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:46,294 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:55:46,294 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:46,294 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:55:46,294 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:46,394 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:55:46,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:55:46,396 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:55:46,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:55:46,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:55:46,397 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:55:46,397 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:46,397 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:55:46,397 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:55:46,397 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:46,397 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:46,398 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:46,398 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:46,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:46,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:46,398 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:46,398 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:55:46,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:46,399 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:46,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:55:46,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:46,399 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:55:46,399 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:55:46,399 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:55:46,399 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:55:46,399 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:46,399 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:55:46,399 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:55:46,399 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:55:46,399 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:46,399 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. started... 2023-03-21 11:55:46,400 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. as already flushing 2023-03-21 11:55:46,400 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:46,400 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:55:46,400 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:55:46,400 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:46,400 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:55:46,400 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:55:46,401 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:46,401 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:46,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:46,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:46,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:46,401 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:46,401 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:55:46,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:46,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:46,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:46,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:46,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:46,402 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:55:46,402 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@654d5696[Count = 0] remaining members to acquire global barrier 2023-03-21 11:55:46,402 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:55:46,402 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:46,403 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:46,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:46,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:46,403 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:55:46,403 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:55:46,403 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:46,403 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:55:46,403 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:55:46,404 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:46,404 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:55:46,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:46,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:46,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:46,404 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:46,404 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:55:46,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:46,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:46,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:46,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:46,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:46,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:46,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:46,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:55:46,406 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:55:46,406 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:55:46,406 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:55:46,406 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:55:46,406 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:55:46,406 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:55:46,409 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:46,409 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:46,409 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:46,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:46,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:46,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:46,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:46,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:46,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:46,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:46,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:46,409 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:46,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:46,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:46,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:46,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:46,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:46,411 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:46,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:46,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:46,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:46,412 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:46,412 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:46,412 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:46,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:46,412 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:46,412 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:55:46,412 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:46,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:46,412 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:46,412 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:55:46,412 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:46,413 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:46,413 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:46,413 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:46,413 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:55:46,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:46,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:46,413 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:55:46,425 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:46,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54894 deadline: 1679399806425, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:46,427 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:46,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54864 deadline: 1679399806426, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:46,446 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:46,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54892 deadline: 1679399806444, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:46,513 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:55:46,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:55:46,515 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:55:46,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:55:46,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:55:46,516 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:55:46,516 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:46,516 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:55:46,516 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:55:46,516 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:46,516 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:46,517 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:46,517 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:46,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:46,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:46,517 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:46,517 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:55:46,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:46,517 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:46,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:55:46,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:46,518 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:55:46,518 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:55:46,518 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:55:46,518 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:55:46,518 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:46,518 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:55:46,518 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:55:46,518 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:55:46,518 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:46,519 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. started... 2023-03-21 11:55:46,519 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. as already flushing 2023-03-21 11:55:46,519 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:46,519 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:55:46,519 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:55:46,519 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:46,519 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:55:46,519 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:55:46,520 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:46,520 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:46,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:46,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:46,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:46,520 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:46,520 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:55:46,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:46,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:46,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:46,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:46,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:46,521 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:55:46,521 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:55:46,521 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ff40f0[Count = 0] remaining members to acquire global barrier 2023-03-21 11:55:46,521 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:46,522 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:46,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:46,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:46,522 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:55:46,522 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:46,522 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:55:46,522 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:55:46,522 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:55:46,523 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:46,523 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:55:46,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:46,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:46,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:46,523 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:46,523 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:55:46,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:46,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:46,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:46,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:46,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:46,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:46,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:46,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:55:46,525 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:55:46,525 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:55:46,525 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:55:46,525 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:55:46,525 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:55:46,525 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:55:46,526 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:46,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:46,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:46,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:46,526 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:46,526 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:46,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:46,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:46,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:46,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:46,526 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:46,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:46,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:46,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:46,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:46,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:46,527 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:46,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:46,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:46,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:46,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:46,529 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:46,529 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:46,529 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:46,529 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:46,529 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:46,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:46,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:46,530 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:46,530 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:46,530 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:46,530 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:46,530 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:55:46,530 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:55:46,530 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:55:46,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:46,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:46,530 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:55:46,530 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:55:46,530 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:46,535 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=830 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/C/6db913c5a4c2484a995871fb7be47e9d 2023-03-21 11:55:46,541 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/A/4fbc9568fd394997823c8df520ccf30f as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/4fbc9568fd394997823c8df520ccf30f 2023-03-21 11:55:46,546 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/4fbc9568fd394997823c8df520ccf30f, entries=300, sequenceid=830, filesize=19.0 K 2023-03-21 11:55:46,547 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/B/ae65b0e384ad4358a63dc9ab53d5e9f7 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/ae65b0e384ad4358a63dc9ab53d5e9f7 2023-03-21 11:55:46,552 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/ae65b0e384ad4358a63dc9ab53d5e9f7, entries=150, sequenceid=830, filesize=11.9 K 2023-03-21 11:55:46,553 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/C/6db913c5a4c2484a995871fb7be47e9d as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/6db913c5a4c2484a995871fb7be47e9d 2023-03-21 11:55:46,560 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/6db913c5a4c2484a995871fb7be47e9d, entries=150, sequenceid=830, filesize=11.9 K 2023-03-21 11:55:46,561 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 1baade11f622687e2c1237b8441afd49 in 1300ms, sequenceid=830, compaction requested=true 2023-03-21 11:55:46,561 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1baade11f622687e2c1237b8441afd49: 2023-03-21 11:55:46,561 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-21 11:55:46,561 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-21 11:55:46,561 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-21 11:55:46,561 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-21 11:55:46,561 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-21 11:55:46,563 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 80676 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-21 11:55:46,563 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1912): 1baade11f622687e2c1237b8441afd49/A is initiating minor compaction (all files) 2023-03-21 11:55:46,564 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1baade11f622687e2c1237b8441afd49/A in TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:46,564 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/dcee3246ab7c4e6293e9f95772fa76bd, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/ae55ceef96844e07b1362851cdc0d7a0, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/6eb0e3139e9743119d8eb5541ba69c5c, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/7520b7ea1bbf4a8d8b901fb9700d9d48, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/3e450e6d72af4f6f8a153fea77e3623c, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/4fbc9568fd394997823c8df520ccf30f] into tmpdir=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp, totalSize=78.8 K 2023-03-21 11:55:46,564 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 73356 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-21 11:55:46,564 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1912): 1baade11f622687e2c1237b8441afd49/B is initiating minor compaction (all files) 2023-03-21 11:55:46,564 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 1baade11f622687e2c1237b8441afd49/B in TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:46,564 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/a9e65d306c2d439d8adece1adacca899, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/e6fbe49c71ea4a30b61e71867fb7a879, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/d93b8a06415240c99c27a1613c713c3f, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/abcc8dbaa4504506ac0769fec5df6e44, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/83de8f130b27469f95deb08d73b65e0e, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/ae65b0e384ad4358a63dc9ab53d5e9f7] into tmpdir=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp, totalSize=71.6 K 2023-03-21 11:55:46,564 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting dcee3246ab7c4e6293e9f95772fa76bd, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=723, earliestPutTs=1679399739720 2023-03-21 11:55:46,566 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting a9e65d306c2d439d8adece1adacca899, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=723, earliestPutTs=1679399739720 2023-03-21 11:55:46,566 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting ae55ceef96844e07b1362851cdc0d7a0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=733, earliestPutTs=1679399740886 2023-03-21 11:55:46,567 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting e6fbe49c71ea4a30b61e71867fb7a879, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=733, earliestPutTs=1679399740886 2023-03-21 11:55:46,567 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 6eb0e3139e9743119d8eb5541ba69c5c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=771, earliestPutTs=1679399742027 2023-03-21 11:55:46,567 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting d93b8a06415240c99c27a1613c713c3f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=771, earliestPutTs=1679399742027 2023-03-21 11:55:46,567 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 7520b7ea1bbf4a8d8b901fb9700d9d48, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=781, earliestPutTs=1679399742660 2023-03-21 11:55:46,567 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting abcc8dbaa4504506ac0769fec5df6e44, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=781, earliestPutTs=1679399742660 2023-03-21 11:55:46,568 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 3e450e6d72af4f6f8a153fea77e3623c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=817, earliestPutTs=1679399743811 2023-03-21 11:55:46,568 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 83de8f130b27469f95deb08d73b65e0e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=817, earliestPutTs=1679399743811 2023-03-21 11:55:46,568 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 4fbc9568fd394997823c8df520ccf30f, keycount=300, bloomtype=ROW, size=19.0 K, encoding=NONE, compression=NONE, seqNum=830, earliestPutTs=1679399744942 2023-03-21 11:55:46,568 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting ae65b0e384ad4358a63dc9ab53d5e9f7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=830, earliestPutTs=1679399744948 2023-03-21 11:55:46,591 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] throttle.PressureAwareThroughputController(145): 1baade11f622687e2c1237b8441afd49#A#compaction#39 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-21 11:55:46,591 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] throttle.PressureAwareThroughputController(145): 1baade11f622687e2c1237b8441afd49#B#compaction#40 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-21 11:55:46,630 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:55:46,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:55:46,633 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:55:46,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:55:46,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:55:46,634 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:55:46,634 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:46,634 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:55:46,634 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:55:46,634 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:46,634 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:46,635 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:46,635 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:46,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:46,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:46,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:46,635 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:46,635 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:55:46,637 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:46,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:55:46,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:46,637 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:55:46,637 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:55:46,637 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:55:46,637 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:55:46,637 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:46,637 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:55:46,637 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:55:46,637 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:55:46,637 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:46,638 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. started... 2023-03-21 11:55:46,638 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1baade11f622687e2c1237b8441afd49 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-21 11:55:46,638 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1baade11f622687e2c1237b8441afd49, store=A 2023-03-21 11:55:46,638 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:55:46,638 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1baade11f622687e2c1237b8441afd49, store=B 2023-03-21 11:55:46,638 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:55:46,638 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1baade11f622687e2c1237b8441afd49, store=C 2023-03-21 11:55:46,638 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:55:47,004 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/B/a32bbbc70d54418188b27b433a4885a2 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/a32bbbc70d54418188b27b433a4885a2 2023-03-21 11:55:47,006 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/A/b6a8ecdb2ea64b988803ff9a0d55541a as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/b6a8ecdb2ea64b988803ff9a0d55541a 2023-03-21 11:55:47,011 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 1baade11f622687e2c1237b8441afd49/B of 1baade11f622687e2c1237b8441afd49 into a32bbbc70d54418188b27b433a4885a2(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-21 11:55:47,015 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1baade11f622687e2c1237b8441afd49: 2023-03-21 11:55:47,015 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49., storeName=1baade11f622687e2c1237b8441afd49/B, priority=10, startTime=1679399746561; duration=0sec 2023-03-21 11:55:47,015 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-21 11:55:47,015 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-21 11:55:47,017 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 1baade11f622687e2c1237b8441afd49/A of 1baade11f622687e2c1237b8441afd49 into b6a8ecdb2ea64b988803ff9a0d55541a(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-21 11:55:47,018 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1baade11f622687e2c1237b8441afd49: 2023-03-21 11:55:47,018 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49., storeName=1baade11f622687e2c1237b8441afd49/A, priority=10, startTime=1679399746561; duration=0sec 2023-03-21 11:55:47,018 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-21 11:55:47,020 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 73356 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-21 11:55:47,020 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1912): 1baade11f622687e2c1237b8441afd49/C is initiating minor compaction (all files) 2023-03-21 11:55:47,020 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 1baade11f622687e2c1237b8441afd49/C in TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:47,020 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/82afd4b6f2a846a9af5f125f12c5fc45, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/dbca2fc359834f2dbe37ff32cf1578eb, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/b76e30e438d0471e999a4b13dde5d7f6, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/5ff8d939f3494163bf8bbe3f10e94e84, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/45d08817049145c087900115258424d4, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/6db913c5a4c2484a995871fb7be47e9d] into tmpdir=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp, totalSize=71.6 K 2023-03-21 11:55:47,021 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 82afd4b6f2a846a9af5f125f12c5fc45, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=723, earliestPutTs=1679399739720 2023-03-21 11:55:47,021 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting dbca2fc359834f2dbe37ff32cf1578eb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=733, earliestPutTs=1679399740886 2023-03-21 11:55:47,022 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting b76e30e438d0471e999a4b13dde5d7f6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=771, earliestPutTs=1679399742027 2023-03-21 11:55:47,022 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 5ff8d939f3494163bf8bbe3f10e94e84, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=781, earliestPutTs=1679399742660 2023-03-21 11:55:47,023 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 45d08817049145c087900115258424d4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=817, earliestPutTs=1679399743811 2023-03-21 11:55:47,023 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 6db913c5a4c2484a995871fb7be47e9d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=830, earliestPutTs=1679399744948 2023-03-21 11:55:47,039 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] throttle.PressureAwareThroughputController(145): 1baade11f622687e2c1237b8441afd49#C#compaction#42 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-21 11:55:47,050 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=849 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/A/d0e42e5a19294e20a53f3d1dba643fe3 2023-03-21 11:55:47,074 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/C/480e1800afbc43c79262c58c534461fd as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/480e1800afbc43c79262c58c534461fd 2023-03-21 11:55:47,084 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 1baade11f622687e2c1237b8441afd49/C of 1baade11f622687e2c1237b8441afd49 into 480e1800afbc43c79262c58c534461fd(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-21 11:55:47,084 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1baade11f622687e2c1237b8441afd49: 2023-03-21 11:55:47,084 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49., storeName=1baade11f622687e2c1237b8441afd49/C, priority=10, startTime=1679399746561; duration=0sec 2023-03-21 11:55:47,084 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-21 11:55:47,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(9158): Flush requested on 1baade11f622687e2c1237b8441afd49 2023-03-21 11:55:47,433 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. as already flushing 2023-03-21 11:55:47,447 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:47,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54894 deadline: 1679399807445, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:47,447 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:47,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54864 deadline: 1679399807446, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:47,457 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:47,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54892 deadline: 1679399807456, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:47,497 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=849 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/B/c4e5686cd5d4489b9a934e20f9b3827b 2023-03-21 11:55:47,519 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=849 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/C/5e51e46a266d459980b796be1813f2ff 2023-03-21 11:55:47,525 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/A/d0e42e5a19294e20a53f3d1dba643fe3 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/d0e42e5a19294e20a53f3d1dba643fe3 2023-03-21 11:55:47,532 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/d0e42e5a19294e20a53f3d1dba643fe3, entries=150, sequenceid=849, filesize=11.9 K 2023-03-21 11:55:47,533 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/B/c4e5686cd5d4489b9a934e20f9b3827b as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/c4e5686cd5d4489b9a934e20f9b3827b 2023-03-21 11:55:47,540 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/c4e5686cd5d4489b9a934e20f9b3827b, entries=150, sequenceid=849, filesize=11.9 K 2023-03-21 11:55:47,541 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/C/5e51e46a266d459980b796be1813f2ff as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/5e51e46a266d459980b796be1813f2ff 2023-03-21 11:55:47,547 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/5e51e46a266d459980b796be1813f2ff, entries=150, sequenceid=849, filesize=11.9 K 2023-03-21 11:55:47,548 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 1baade11f622687e2c1237b8441afd49 in 910ms, sequenceid=849, compaction requested=false 2023-03-21 11:55:47,548 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1baade11f622687e2c1237b8441afd49: 2023-03-21 11:55:47,548 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:47,548 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:55:47,548 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:55:47,549 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:47,550 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:55:47,550 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:55:47,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(9158): Flush requested on 1baade11f622687e2c1237b8441afd49 2023-03-21 11:55:47,551 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1baade11f622687e2c1237b8441afd49 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-03-21 11:55:47,551 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:47,551 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:47,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:47,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:47,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:47,552 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:47,552 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:55:47,552 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1baade11f622687e2c1237b8441afd49, store=A 2023-03-21 11:55:47,552 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:55:47,552 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1baade11f622687e2c1237b8441afd49, store=B 2023-03-21 11:55:47,552 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:55:47,552 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1baade11f622687e2c1237b8441afd49, store=C 2023-03-21 11:55:47,552 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:55:47,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:47,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:47,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:47,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:47,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:47,555 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:55:47,555 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@39ef4655[Count = 0] remaining members to acquire global barrier 2023-03-21 11:55:47,555 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:55:47,556 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:47,557 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:47,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:47,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:47,557 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:55:47,557 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:55:47,557 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:55:47,557 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:47,557 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:55:47,560 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:47,560 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:55:47,560 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:47,560 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:55:47,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:47,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:47,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:47,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:47,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:47,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:47,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:47,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:47,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:47,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:47,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:55:47,564 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:55:47,564 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:55:47,564 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:55:47,564 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:55:47,564 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:55:47,564 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:55:47,571 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:47,571 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:47,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:47,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:47,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:47,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:47,571 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:47,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:47,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:47,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:47,572 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:47,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:47,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:47,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:47,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:47,573 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:47,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:47,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:47,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:47,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:47,581 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:47,581 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:47,581 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:47,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:47,581 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:47,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:47,581 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:47,581 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:47,581 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:47,582 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:47,582 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:55:47,582 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:55:47,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:47,582 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:47,582 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:47,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:47,582 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:55:47,582 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:55:47,582 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:55:47,597 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:47,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54894 deadline: 1679399807594, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:47,599 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:47,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54864 deadline: 1679399807597, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:47,682 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:55:47,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:55:47,684 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:55:47,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:55:47,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:55:47,685 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:55:47,685 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:47,685 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:55:47,686 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:55:47,686 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:47,686 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:47,687 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:47,687 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:47,687 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:55:47,687 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:47,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:47,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:47,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:47,688 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:47,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:55:47,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:47,688 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:55:47,688 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:55:47,688 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:55:47,688 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:55:47,688 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:47,688 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:55:47,688 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:55:47,689 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:55:47,689 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:47,689 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. started... 2023-03-21 11:55:47,689 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. as already flushing 2023-03-21 11:55:47,689 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:47,689 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:55:47,689 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:55:47,689 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:47,689 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:55:47,689 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:55:47,690 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:47,690 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:47,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:47,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:47,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:47,690 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:47,690 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:55:47,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:47,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:47,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:47,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:47,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:47,692 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:55:47,692 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@57b7fdd7[Count = 0] remaining members to acquire global barrier 2023-03-21 11:55:47,692 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:55:47,692 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:47,692 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:47,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:47,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:47,693 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:55:47,693 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:55:47,693 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:55:47,693 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:47,693 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:55:47,694 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:47,694 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:55:47,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:47,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:47,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:47,694 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:47,694 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:55:47,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:47,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:47,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:47,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:47,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:47,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:47,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:47,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:55:47,696 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:55:47,696 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:55:47,696 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:55:47,696 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:55:47,696 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:55:47,696 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:55:47,697 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:47,697 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:47,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:47,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:47,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:47,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:47,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:47,697 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:47,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:47,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:47,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:47,698 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:47,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:47,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:47,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:47,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:47,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:47,699 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:47,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:47,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:47,700 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:47,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54894 deadline: 1679399807699, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:47,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:47,702 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:47,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 246 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54864 deadline: 1679399807701, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:47,702 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:47,702 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:47,702 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:47,702 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:47,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:47,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:47,702 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:55:47,702 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:55:47,702 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:47,702 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:55:47,702 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:47,702 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:47,702 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:55:47,703 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:47,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:47,703 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:55:47,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:47,703 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:47,703 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:47,803 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:55:47,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:55:47,805 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:55:47,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:55:47,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:55:47,806 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:55:47,806 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:47,806 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:55:47,806 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:55:47,806 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:47,806 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:47,807 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:47,807 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:47,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:47,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:47,807 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:47,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:47,807 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:55:47,808 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:47,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:55:47,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:47,808 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:55:47,808 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:55:47,808 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:55:47,808 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:55:47,808 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:47,808 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:55:47,809 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:55:47,809 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:55:47,809 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:47,809 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. started... 2023-03-21 11:55:47,809 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. as already flushing 2023-03-21 11:55:47,809 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:47,809 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:55:47,809 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:55:47,809 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:47,809 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:55:47,809 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:55:47,810 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:47,810 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:47,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:47,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:47,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:47,810 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:47,810 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:55:47,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:47,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:47,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:47,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:47,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:47,811 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:55:47,811 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:55:47,811 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@58c1f851[Count = 0] remaining members to acquire global barrier 2023-03-21 11:55:47,811 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:47,812 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:47,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:47,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:47,812 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:55:47,812 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:55:47,812 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:55:47,812 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:47,812 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:55:47,813 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:55:47,813 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:47,813 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:47,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:47,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:47,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:47,813 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:55:47,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:47,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:47,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:47,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:47,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:47,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:47,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:47,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:55:47,815 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:55:47,815 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:55:47,815 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:55:47,815 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:55:47,815 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:55:47,815 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:55:47,816 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:47,816 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:47,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:47,816 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:47,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:47,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:47,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:47,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:47,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:47,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:47,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:47,817 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:47,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:47,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:47,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:47,818 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:47,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:47,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:47,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:47,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:47,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:47,819 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:47,819 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:47,819 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:47,819 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:55:47,819 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:55:47,819 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:47,820 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:47,820 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:47,820 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:47,819 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:47,819 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:47,820 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:55:47,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:47,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:47,820 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:55:47,820 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:55:47,820 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:47,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:47,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:47,902 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:47,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54894 deadline: 1679399807902, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:47,906 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:47,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 248 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54864 deadline: 1679399807904, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:47,920 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:55:47,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:55:47,922 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:55:47,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:55:47,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:55:47,923 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:55:47,923 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:47,923 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:55:47,923 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:55:47,924 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:47,924 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:47,924 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:47,924 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:47,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:47,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:47,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:47,925 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:47,925 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:55:47,925 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:47,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:55:47,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:47,925 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:55:47,926 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:55:47,926 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:55:47,926 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:55:47,926 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:47,926 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:55:47,926 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:55:47,926 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:55:47,926 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:47,926 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. started... 2023-03-21 11:55:47,926 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. as already flushing 2023-03-21 11:55:47,926 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:47,926 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:55:47,926 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:55:47,926 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:47,926 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:55:47,926 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:55:47,927 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:47,927 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:47,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:47,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:47,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:47,927 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:47,927 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:55:47,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:47,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:47,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:47,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:47,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:47,928 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:55:47,928 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@21c0cb79[Count = 0] remaining members to acquire global barrier 2023-03-21 11:55:47,928 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:55:47,929 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:47,929 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:47,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:47,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:47,929 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:55:47,929 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:55:47,929 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:55:47,929 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:47,929 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:55:47,930 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:47,930 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:55:47,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:47,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:47,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:47,930 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:47,930 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:55:47,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:47,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:47,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:47,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:47,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:47,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:47,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:47,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:55:47,932 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:55:47,932 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:55:47,932 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:55:47,932 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:55:47,932 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:55:47,932 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:55:47,933 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:47,933 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:47,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:47,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:47,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:47,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:47,933 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:47,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:47,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:47,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:47,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:47,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:47,934 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:47,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:47,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:47,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:47,935 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:47,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:47,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:47,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:47,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:47,936 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:47,936 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:47,936 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:47,936 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:47,936 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:47,936 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:47,936 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:47,937 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:47,937 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:47,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:47,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:47,937 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:55:47,937 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:55:47,937 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:47,937 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:55:47,937 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:55:47,937 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:55:47,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:47,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:47,964 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=867 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/A/994a302de1c44da3beccbb1e98a25514 2023-03-21 11:55:47,974 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:47,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54840 deadline: 1679399807973, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:47,975 DEBUG [Thread-143] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4158 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49., hostname=jenkins-hbase20.apache.org,34369,1679399732030, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-21 11:55:47,980 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:47,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54904 deadline: 1679399807977, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:47,982 DEBUG [Thread-149] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4164 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49., hostname=jenkins-hbase20.apache.org,34369,1679399732030, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-21 11:55:48,037 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:55:48,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:55:48,039 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:55:48,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:55:48,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:55:48,040 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:55:48,040 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:48,040 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:55:48,040 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:55:48,041 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:48,041 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:48,042 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:48,042 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:48,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:48,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:48,042 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:48,042 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:55:48,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:48,042 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:48,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:55:48,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:48,043 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:55:48,043 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:55:48,043 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:55:48,043 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:55:48,043 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:48,043 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:55:48,043 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:55:48,043 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:55:48,043 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:48,044 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. started... 2023-03-21 11:55:48,044 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. as already flushing 2023-03-21 11:55:48,044 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:48,044 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:55:48,044 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:55:48,044 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:48,044 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:55:48,044 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:55:48,045 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:48,045 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:48,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:48,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:48,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:48,045 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:48,045 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:55:48,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:48,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:48,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:48,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:48,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:48,046 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:55:48,046 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@62deecfa[Count = 0] remaining members to acquire global barrier 2023-03-21 11:55:48,046 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:55:48,047 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:48,050 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:48,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:48,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:48,051 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:55:48,051 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:48,051 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:55:48,051 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:55:48,051 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:55:48,053 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:48,053 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:55:48,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:48,053 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:48,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:48,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:48,053 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:55:48,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:48,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:48,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:48,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:48,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:48,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:48,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:48,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:55:48,056 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:55:48,056 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:55:48,056 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:55:48,056 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:55:48,056 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:55:48,056 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:55:48,057 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:48,057 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:48,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:48,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:48,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:48,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:48,057 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:48,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:48,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:48,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:48,057 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:48,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:48,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:48,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:48,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:48,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:48,058 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:48,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:48,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:48,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:48,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:48,061 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:48,061 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:48,061 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:48,061 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:55:48,061 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:55:48,061 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:55:48,061 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:48,061 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:48,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:48,061 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:48,061 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:55:48,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:48,061 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:48,061 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:55:48,061 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:48,061 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:48,061 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:48,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:48,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:48,161 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:55:48,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:55:48,163 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:55:48,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:55:48,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:55:48,164 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:55:48,164 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:48,164 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:55:48,164 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:55:48,165 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:48,165 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:48,165 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:48,165 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:48,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:48,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:48,166 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:48,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:48,166 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:55:48,166 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:48,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:55:48,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:48,166 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:55:48,167 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:55:48,167 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:55:48,167 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:55:48,167 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:48,167 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:55:48,167 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:55:48,167 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:55:48,167 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:48,167 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. started... 2023-03-21 11:55:48,167 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. as already flushing 2023-03-21 11:55:48,167 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:48,167 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:55:48,167 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:55:48,167 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:48,167 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:55:48,167 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:55:48,168 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:48,168 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:48,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:48,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:48,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:48,169 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:48,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:48,169 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:55:48,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:48,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:48,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:48,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:48,170 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:55:48,170 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@aeffc33[Count = 0] remaining members to acquire global barrier 2023-03-21 11:55:48,170 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:55:48,170 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:48,171 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:48,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:48,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:48,171 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:55:48,171 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:55:48,171 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:48,171 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:55:48,171 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:55:48,172 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:48,172 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:55:48,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:48,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:48,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:48,172 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:48,172 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:55:48,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:48,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:48,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:48,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:48,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:48,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:48,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:48,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:55:48,174 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:55:48,174 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:55:48,174 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:55:48,174 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:55:48,174 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:55:48,174 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:55:48,175 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:48,175 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:48,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:48,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:48,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:48,175 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:48,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:48,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:48,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:48,176 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:48,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:48,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:48,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:48,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:48,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:48,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:48,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:48,177 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:48,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:48,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:48,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:48,179 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:48,179 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:48,179 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:48,179 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:48,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:48,179 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:48,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:48,179 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:55:48,179 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:55:48,179 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:48,179 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:48,179 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:48,179 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:55:48,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:48,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:48,179 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:48,179 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:55:48,180 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:48,207 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:48,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54894 deadline: 1679399808205, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:48,209 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:48,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 250 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54864 deadline: 1679399808207, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:48,280 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:55:48,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:55:48,282 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:55:48,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:55:48,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:55:48,294 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:55:48,294 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:48,294 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:55:48,294 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:55:48,295 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:48,295 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:48,296 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:48,296 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:48,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:48,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:48,296 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:48,296 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:55:48,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:48,297 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:48,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:55:48,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:48,297 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:55:48,298 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:55:48,298 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:55:48,298 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:55:48,298 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:48,298 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:55:48,298 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:55:48,298 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:55:48,298 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:48,298 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. started... 2023-03-21 11:55:48,298 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. as already flushing 2023-03-21 11:55:48,298 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:48,298 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:55:48,298 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:55:48,298 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:48,299 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:55:48,299 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:55:48,306 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:48,306 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:48,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:48,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:48,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:48,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:48,306 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:48,307 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:55:48,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:48,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:48,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:48,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:48,308 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:55:48,308 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:55:48,308 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5dbb8e4f[Count = 0] remaining members to acquire global barrier 2023-03-21 11:55:48,308 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:48,308 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:48,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:48,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:48,309 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:55:48,309 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:55:48,309 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:48,309 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:55:48,309 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:55:48,310 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:48,310 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:55:48,310 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:48,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:48,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:48,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:48,310 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:55:48,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:48,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:48,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:48,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:48,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:48,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:48,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:48,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:55:48,312 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:55:48,312 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:55:48,312 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:55:48,312 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:55:48,312 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:55:48,312 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:55:48,313 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:48,313 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:48,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:48,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:48,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:48,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:48,313 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:48,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:48,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:48,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:48,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:48,314 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:48,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:48,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:48,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:48,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:48,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:48,315 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:48,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:48,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:48,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:48,317 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:48,317 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:48,317 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:48,317 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:48,317 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:48,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:48,317 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:48,317 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:48,317 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:48,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:48,317 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:55:48,317 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:55:48,317 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:48,317 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:48,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:48,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:48,317 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:55:48,317 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:55:48,378 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=867 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/B/ddb7e67da9ac4fa594352481b78512c1 2023-03-21 11:55:48,417 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:55:48,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:55:48,420 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:55:48,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:55:48,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:55:48,421 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:55:48,421 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:48,421 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:55:48,421 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:55:48,421 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:48,421 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:48,422 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:48,422 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:48,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:48,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:48,423 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:48,423 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:55:48,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:48,423 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:48,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:55:48,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:48,423 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:55:48,424 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:55:48,424 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:55:48,424 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:55:48,424 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:48,424 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:55:48,424 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:55:48,424 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:55:48,424 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:48,424 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. started... 2023-03-21 11:55:48,424 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. as already flushing 2023-03-21 11:55:48,424 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:48,425 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:55:48,425 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:55:48,425 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:48,425 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:55:48,425 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:55:48,426 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:48,426 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:48,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:48,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:48,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:48,427 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:48,427 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:55:48,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:48,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:48,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:48,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:48,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:48,428 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:55:48,428 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a871d82[Count = 0] remaining members to acquire global barrier 2023-03-21 11:55:48,428 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:55:48,428 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:48,429 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:48,429 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:48,429 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:55:48,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:48,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:48,429 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:55:48,429 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:55:48,429 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:55:48,431 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:55:48,431 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:48,431 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:48,431 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:55:48,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:48,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:48,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:48,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:48,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:48,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:48,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:48,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:48,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:48,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:48,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:55:48,433 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:55:48,433 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:55:48,433 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:55:48,433 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:55:48,433 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:55:48,433 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:55:48,434 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:48,434 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:48,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:48,434 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:48,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:48,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:48,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:48,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:48,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:48,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:48,434 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:48,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:48,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:48,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:48,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:48,435 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:48,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:48,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:48,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:48,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:48,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:48,437 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:48,437 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:48,437 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:48,437 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:48,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:48,437 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:48,437 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:48,437 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:55:48,437 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:55:48,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:48,437 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:48,437 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:55:48,438 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:48,438 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:48,438 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:55:48,438 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:48,438 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:55:48,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:48,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:48,538 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:55:48,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:55:48,540 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:55:48,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:55:48,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:55:48,541 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:55:48,541 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:48,541 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:55:48,541 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:55:48,541 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:48,541 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:48,542 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:48,542 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:48,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:48,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:48,542 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:48,542 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:55:48,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:48,542 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:48,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:55:48,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:48,543 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:55:48,543 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:55:48,543 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:55:48,543 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:55:48,543 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:48,543 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:55:48,543 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:55:48,544 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:55:48,544 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:48,544 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. started... 2023-03-21 11:55:48,544 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. as already flushing 2023-03-21 11:55:48,544 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:48,544 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:55:48,544 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:55:48,544 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:48,544 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:55:48,544 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:55:48,546 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:48,546 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:48,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:48,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:48,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:48,546 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:48,546 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:55:48,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:48,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:48,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:48,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:48,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:48,547 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:55:48,547 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:55:48,547 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6fab8714[Count = 0] remaining members to acquire global barrier 2023-03-21 11:55:48,547 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:48,548 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:48,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:48,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:48,548 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:55:48,548 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:55:48,548 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:55:48,548 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:48,549 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:55:48,549 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:48,549 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:55:48,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:48,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:48,549 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:48,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:48,549 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:55:48,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:48,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:48,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:48,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:48,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:48,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:48,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:48,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:55:48,551 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:55:48,551 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:55:48,552 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:55:48,552 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:55:48,552 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:55:48,552 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:55:48,553 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:48,553 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:48,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:48,553 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:48,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:48,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:48,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:48,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:48,553 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:48,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:48,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:48,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:48,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:48,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:48,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:48,555 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:48,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:48,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:48,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:48,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:48,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:48,558 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:48,558 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:48,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:48,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:48,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:48,559 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:48,559 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:48,559 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:55:48,559 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:55:48,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:48,559 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:48,559 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:48,559 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:55:48,559 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:55:48,559 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:48,559 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:55:48,559 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:48,560 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:48,560 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:48,660 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:55:48,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:55:48,661 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:55:48,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:55:48,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:55:48,662 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:55:48,662 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:48,662 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:55:48,663 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:55:48,663 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:48,663 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:48,664 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:48,664 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:48,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:48,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:48,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:48,664 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:48,664 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:55:48,664 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:48,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:55:48,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:48,664 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:55:48,665 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:55:48,665 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:55:48,665 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:55:48,665 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:48,665 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:55:48,665 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:55:48,665 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:55:48,665 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:48,665 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. started... 2023-03-21 11:55:48,665 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. as already flushing 2023-03-21 11:55:48,665 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:48,665 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:55:48,665 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:55:48,665 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:48,665 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:55:48,665 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:55:48,666 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:48,666 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:48,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:48,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:48,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:48,666 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:48,666 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:55:48,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:48,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:48,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:48,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:48,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:48,667 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:55:48,668 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1ad0e0b6[Count = 0] remaining members to acquire global barrier 2023-03-21 11:55:48,668 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:55:48,668 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:48,668 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:48,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:48,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:48,668 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:55:48,668 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:55:48,668 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:48,668 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:55:48,668 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:55:48,669 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:48,669 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:55:48,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:48,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:48,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:48,669 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:48,669 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:55:48,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:48,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:48,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:48,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:48,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:48,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:48,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:48,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:55:48,671 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:55:48,671 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:55:48,671 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:55:48,671 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:55:48,671 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:55:48,671 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:55:48,673 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:48,673 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:48,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:48,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:48,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:48,673 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:48,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:48,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:48,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:48,674 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:48,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:48,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:48,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:48,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:48,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:48,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:48,676 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:48,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:48,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:48,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:48,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:48,677 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:48,677 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:48,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:48,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:48,677 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:48,677 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:48,677 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:48,677 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:48,677 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:55:48,677 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:55:48,677 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:48,677 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:48,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:48,677 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:48,678 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:55:48,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:48,678 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:55:48,678 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:55:48,678 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:48,713 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:48,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54894 deadline: 1679399808709, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:48,713 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:48,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 252 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54864 deadline: 1679399808711, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:48,778 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:55:48,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:55:48,780 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:55:48,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:55:48,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:55:48,781 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:55:48,781 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:48,781 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:55:48,781 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:55:48,782 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:48,782 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:48,782 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:48,782 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:48,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:48,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:48,782 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:48,783 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:55:48,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:48,783 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:48,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:55:48,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:48,783 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:55:48,783 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:55:48,783 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:55:48,783 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:55:48,783 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:48,783 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:55:48,783 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:55:48,784 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:55:48,784 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:48,784 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. started... 2023-03-21 11:55:48,784 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. as already flushing 2023-03-21 11:55:48,784 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:48,784 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:55:48,784 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:55:48,784 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:48,784 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:55:48,784 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:55:48,785 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:48,785 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:48,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:48,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:48,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:48,785 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:48,785 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:55:48,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:48,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:48,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:48,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:48,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:48,786 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:55:48,786 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:55:48,786 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b98dc55[Count = 0] remaining members to acquire global barrier 2023-03-21 11:55:48,786 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:48,787 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:48,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:48,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:48,787 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:55:48,787 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:55:48,787 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:55:48,787 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:48,787 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:55:48,788 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:48,788 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:55:48,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:48,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:48,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:48,788 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:48,788 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:55:48,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:48,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:48,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:48,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:48,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:48,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:48,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:48,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:55:48,789 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:55:48,789 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:55:48,790 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:55:48,790 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:55:48,790 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:55:48,790 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:55:48,790 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:48,790 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:48,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:48,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:48,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:48,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:48,791 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:48,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:48,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:48,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:48,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:48,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:48,791 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:48,791 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=867 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/C/cdf2cd8db0234c51ad34438134e69923 2023-03-21 11:55:48,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:48,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:48,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:48,793 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:48,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:48,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:48,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:48,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:48,795 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:48,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:48,795 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:48,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:48,795 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:48,795 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:48,795 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:55:48,796 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:55:48,795 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:48,796 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:48,796 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:48,796 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:48,796 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:48,796 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:55:48,796 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:55:48,796 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:55:48,796 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:48,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:48,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:48,798 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/A/994a302de1c44da3beccbb1e98a25514 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/994a302de1c44da3beccbb1e98a25514 2023-03-21 11:55:48,803 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/994a302de1c44da3beccbb1e98a25514, entries=250, sequenceid=867, filesize=16.7 K 2023-03-21 11:55:48,804 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/B/ddb7e67da9ac4fa594352481b78512c1 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/ddb7e67da9ac4fa594352481b78512c1 2023-03-21 11:55:48,810 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/ddb7e67da9ac4fa594352481b78512c1, entries=150, sequenceid=867, filesize=11.9 K 2023-03-21 11:55:48,811 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/C/cdf2cd8db0234c51ad34438134e69923 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/cdf2cd8db0234c51ad34438134e69923 2023-03-21 11:55:48,816 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/cdf2cd8db0234c51ad34438134e69923, entries=150, sequenceid=867, filesize=11.9 K 2023-03-21 11:55:48,817 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 1baade11f622687e2c1237b8441afd49 in 1265ms, sequenceid=867, compaction requested=true 2023-03-21 11:55:48,817 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1baade11f622687e2c1237b8441afd49: 2023-03-21 11:55:48,817 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-21 11:55:48,817 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-21 11:55:48,817 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-21 11:55:48,817 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-21 11:55:48,817 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-21 11:55:48,818 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 41882 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-21 11:55:48,818 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1912): 1baade11f622687e2c1237b8441afd49/A is initiating minor compaction (all files) 2023-03-21 11:55:48,818 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37002 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-21 11:55:48,818 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 1baade11f622687e2c1237b8441afd49/A in TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:48,818 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1912): 1baade11f622687e2c1237b8441afd49/B is initiating minor compaction (all files) 2023-03-21 11:55:48,818 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1baade11f622687e2c1237b8441afd49/B in TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:48,818 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/b6a8ecdb2ea64b988803ff9a0d55541a, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/d0e42e5a19294e20a53f3d1dba643fe3, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/994a302de1c44da3beccbb1e98a25514] into tmpdir=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp, totalSize=40.9 K 2023-03-21 11:55:48,818 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/a32bbbc70d54418188b27b433a4885a2, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/c4e5686cd5d4489b9a934e20f9b3827b, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/ddb7e67da9ac4fa594352481b78512c1] into tmpdir=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp, totalSize=36.1 K 2023-03-21 11:55:48,819 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting b6a8ecdb2ea64b988803ff9a0d55541a, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=830, earliestPutTs=1679399744948 2023-03-21 11:55:48,819 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting a32bbbc70d54418188b27b433a4885a2, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=830, earliestPutTs=1679399744948 2023-03-21 11:55:48,819 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting d0e42e5a19294e20a53f3d1dba643fe3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=849, earliestPutTs=1679399745288 2023-03-21 11:55:48,819 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting c4e5686cd5d4489b9a934e20f9b3827b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=849, earliestPutTs=1679399745288 2023-03-21 11:55:48,820 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting ddb7e67da9ac4fa594352481b78512c1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=867, earliestPutTs=1679399747444 2023-03-21 11:55:48,820 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 994a302de1c44da3beccbb1e98a25514, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=867, earliestPutTs=1679399747441 2023-03-21 11:55:48,828 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] throttle.PressureAwareThroughputController(145): 1baade11f622687e2c1237b8441afd49#B#compaction#48 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-21 11:55:48,828 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] throttle.PressureAwareThroughputController(145): 1baade11f622687e2c1237b8441afd49#A#compaction#49 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-21 11:55:48,896 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:55:48,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:55:48,898 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:55:48,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:55:48,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:55:48,899 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:55:48,899 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:48,899 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:55:48,899 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:55:48,899 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:48,899 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:48,900 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:48,900 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:48,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:48,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:48,900 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:48,900 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:55:48,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:48,901 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:48,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:55:48,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:48,901 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:55:48,901 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:55:48,901 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:55:48,901 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:55:48,901 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:48,901 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:55:48,905 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:55:48,905 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:48,905 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:55:48,905 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. started... 2023-03-21 11:55:48,905 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1baade11f622687e2c1237b8441afd49 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-03-21 11:55:48,906 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1baade11f622687e2c1237b8441afd49, store=A 2023-03-21 11:55:48,906 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:55:48,906 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1baade11f622687e2c1237b8441afd49, store=B 2023-03-21 11:55:48,906 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:55:48,906 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1baade11f622687e2c1237b8441afd49, store=C 2023-03-21 11:55:48,906 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:55:49,239 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/B/9653f09670654169b0bb33dc6a2431a5 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/9653f09670654169b0bb33dc6a2431a5 2023-03-21 11:55:49,241 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/A/33cf0b3361084c078149f0e649518554 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/33cf0b3361084c078149f0e649518554 2023-03-21 11:55:49,246 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1baade11f622687e2c1237b8441afd49/B of 1baade11f622687e2c1237b8441afd49 into 9653f09670654169b0bb33dc6a2431a5(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-21 11:55:49,246 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1baade11f622687e2c1237b8441afd49: 2023-03-21 11:55:49,246 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49., storeName=1baade11f622687e2c1237b8441afd49/B, priority=13, startTime=1679399748817; duration=0sec 2023-03-21 11:55:49,246 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-21 11:55:49,246 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-21 11:55:49,248 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1baade11f622687e2c1237b8441afd49/A of 1baade11f622687e2c1237b8441afd49 into 33cf0b3361084c078149f0e649518554(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-21 11:55:49,248 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1baade11f622687e2c1237b8441afd49: 2023-03-21 11:55:49,248 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49., storeName=1baade11f622687e2c1237b8441afd49/A, priority=13, startTime=1679399748817; duration=0sec 2023-03-21 11:55:49,248 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-21 11:55:49,248 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37002 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-21 11:55:49,248 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1912): 1baade11f622687e2c1237b8441afd49/C is initiating minor compaction (all files) 2023-03-21 11:55:49,248 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1baade11f622687e2c1237b8441afd49/C in TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:49,248 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/480e1800afbc43c79262c58c534461fd, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/5e51e46a266d459980b796be1813f2ff, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/cdf2cd8db0234c51ad34438134e69923] into tmpdir=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp, totalSize=36.1 K 2023-03-21 11:55:49,249 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 480e1800afbc43c79262c58c534461fd, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=830, earliestPutTs=1679399744948 2023-03-21 11:55:49,249 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 5e51e46a266d459980b796be1813f2ff, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=849, earliestPutTs=1679399745288 2023-03-21 11:55:49,250 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting cdf2cd8db0234c51ad34438134e69923, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=867, earliestPutTs=1679399747444 2023-03-21 11:55:49,256 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] throttle.PressureAwareThroughputController(145): 1baade11f622687e2c1237b8441afd49#C#compaction#51 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-21 11:55:49,315 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=884 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/A/cbc30cb2011d42d78d763b7fcb2fcba6 2023-03-21 11:55:49,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(9158): Flush requested on 1baade11f622687e2c1237b8441afd49 2023-03-21 11:55:49,465 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. as already flushing 2023-03-21 11:55:49,494 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:49,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54892 deadline: 1679399809491, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:49,597 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:49,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54892 deadline: 1679399809596, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:49,668 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/C/7f32ec67ab3f4ec8afadfc498bc63068 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/7f32ec67ab3f4ec8afadfc498bc63068 2023-03-21 11:55:49,676 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1baade11f622687e2c1237b8441afd49/C of 1baade11f622687e2c1237b8441afd49 into 7f32ec67ab3f4ec8afadfc498bc63068(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-21 11:55:49,676 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1baade11f622687e2c1237b8441afd49: 2023-03-21 11:55:49,676 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49., storeName=1baade11f622687e2c1237b8441afd49/C, priority=13, startTime=1679399748817; duration=0sec 2023-03-21 11:55:49,676 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-21 11:55:49,722 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:49,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 254 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54864 deadline: 1679399809721, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:49,722 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:49,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54894 deadline: 1679399809721, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:49,729 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=884 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/B/b7286c70fb974fee938b24e92badf461 2023-03-21 11:55:49,803 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:49,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54892 deadline: 1679399809800, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:50,104 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:50,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54892 deadline: 1679399810104, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:50,146 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=884 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/C/f9de91cdac0944069e36c00cb76e0a3b 2023-03-21 11:55:50,156 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/A/cbc30cb2011d42d78d763b7fcb2fcba6 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/cbc30cb2011d42d78d763b7fcb2fcba6 2023-03-21 11:55:50,163 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/cbc30cb2011d42d78d763b7fcb2fcba6, entries=150, sequenceid=884, filesize=11.9 K 2023-03-21 11:55:50,165 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/B/b7286c70fb974fee938b24e92badf461 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/b7286c70fb974fee938b24e92badf461 2023-03-21 11:55:50,175 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/b7286c70fb974fee938b24e92badf461, entries=150, sequenceid=884, filesize=11.9 K 2023-03-21 11:55:50,176 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/C/f9de91cdac0944069e36c00cb76e0a3b as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/f9de91cdac0944069e36c00cb76e0a3b 2023-03-21 11:55:50,184 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/f9de91cdac0944069e36c00cb76e0a3b, entries=150, sequenceid=884, filesize=11.9 K 2023-03-21 11:55:50,184 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 1baade11f622687e2c1237b8441afd49 in 1279ms, sequenceid=884, compaction requested=false 2023-03-21 11:55:50,184 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1baade11f622687e2c1237b8441afd49: 2023-03-21 11:55:50,184 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:50,184 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:55:50,184 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:55:50,184 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:50,185 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:55:50,185 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:55:50,186 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:50,186 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:50,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:50,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:50,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:50,186 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:50,186 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:55:50,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:50,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:50,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:50,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:50,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:50,187 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:55:50,188 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5931a9a9[Count = 0] remaining members to acquire global barrier 2023-03-21 11:55:50,188 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:55:50,188 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:50,189 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:50,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:50,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:50,189 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:50,189 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:55:50,189 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:55:50,190 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:55:50,190 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:55:50,190 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:55:50,190 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:50,190 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:50,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:50,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:50,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:50,190 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:55:50,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:50,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:50,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:50,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:50,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:50,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:50,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:50,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:55:50,193 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:55:50,193 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:55:50,193 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:55:50,193 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:55:50,193 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:55:50,193 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:55:50,194 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:50,195 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:50,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:50,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:50,195 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:50,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:50,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:50,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:50,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:50,195 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:50,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:50,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:50,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:50,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:50,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:50,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:50,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:50,197 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:50,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:50,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:50,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:50,199 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:50,199 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:50,199 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:50,199 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:50,199 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:50,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:50,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:50,199 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:50,199 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:55:50,199 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:55:50,199 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:55:50,199 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:50,199 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:50,199 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:50,200 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:50,200 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:55:50,200 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:55:50,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:50,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:50,300 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:55:50,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:55:50,301 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:55:50,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:55:50,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:55:50,302 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:55:50,302 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:50,302 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:55:50,302 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:55:50,303 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:50,303 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:50,303 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:50,303 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:50,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:50,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:50,304 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:50,304 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:55:50,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:50,304 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:50,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:55:50,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:50,304 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:55:50,305 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:55:50,305 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:55:50,305 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:55:50,305 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:50,305 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:55:50,305 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:55:50,305 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:55:50,305 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:50,305 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. started... 2023-03-21 11:55:50,305 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1baade11f622687e2c1237b8441afd49 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-03-21 11:55:50,305 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1baade11f622687e2c1237b8441afd49, store=A 2023-03-21 11:55:50,305 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:55:50,305 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1baade11f622687e2c1237b8441afd49, store=B 2023-03-21 11:55:50,306 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:55:50,306 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1baade11f622687e2c1237b8441afd49, store=C 2023-03-21 11:55:50,306 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:55:50,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(9158): Flush requested on 1baade11f622687e2c1237b8441afd49 2023-03-21 11:55:50,609 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. as already flushing 2023-03-21 11:55:50,634 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:50,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54892 deadline: 1679399810633, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:50,714 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=902 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/A/00f15a59a24247b8b61cd5360a185ee8 2023-03-21 11:55:50,736 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:50,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54892 deadline: 1679399810735, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:50,941 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:50,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54892 deadline: 1679399810939, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:51,136 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=902 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/B/9b1dfc4ceaf7414aba148680881a7954 2023-03-21 11:55:51,244 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:51,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54892 deadline: 1679399811243, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:51,547 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=902 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/C/5193289533354700ada1dddf509941dd 2023-03-21 11:55:51,552 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/A/00f15a59a24247b8b61cd5360a185ee8 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/00f15a59a24247b8b61cd5360a185ee8 2023-03-21 11:55:51,557 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/00f15a59a24247b8b61cd5360a185ee8, entries=150, sequenceid=902, filesize=11.9 K 2023-03-21 11:55:51,558 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/B/9b1dfc4ceaf7414aba148680881a7954 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/9b1dfc4ceaf7414aba148680881a7954 2023-03-21 11:55:51,563 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/9b1dfc4ceaf7414aba148680881a7954, entries=150, sequenceid=902, filesize=11.9 K 2023-03-21 11:55:51,564 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/C/5193289533354700ada1dddf509941dd as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/5193289533354700ada1dddf509941dd 2023-03-21 11:55:51,568 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/5193289533354700ada1dddf509941dd, entries=150, sequenceid=902, filesize=11.9 K 2023-03-21 11:55:51,569 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 1baade11f622687e2c1237b8441afd49 in 1264ms, sequenceid=902, compaction requested=true 2023-03-21 11:55:51,569 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1baade11f622687e2c1237b8441afd49: 2023-03-21 11:55:51,569 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:51,569 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:55:51,569 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:55:51,569 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:51,569 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:55:51,569 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:55:51,570 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:51,570 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:51,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:51,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:51,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:51,570 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:51,570 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:55:51,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:51,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:51,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:51,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:51,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:51,571 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:55:51,571 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a9116bc[Count = 0] remaining members to acquire global barrier 2023-03-21 11:55:51,571 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:55:51,572 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:51,572 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:51,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:51,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:51,572 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:55:51,572 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:55:51,572 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:55:51,572 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:51,572 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:55:51,573 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:55:51,573 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:51,573 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:51,573 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:55:51,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:51,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:51,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:51,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:51,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:51,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:51,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:51,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:51,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:51,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:51,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:55:51,575 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:55:51,575 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:55:51,575 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:55:51,576 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:55:51,576 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:55:51,576 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:55:51,576 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:51,576 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:51,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:51,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:51,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:51,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:51,577 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:51,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:51,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:51,577 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:51,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:51,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:51,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:51,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:51,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:51,578 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:51,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:51,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:51,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:51,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:51,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:51,580 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:51,580 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:51,580 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:51,580 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:51,580 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:51,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:51,580 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:55:51,580 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:55:51,580 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:51,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:51,580 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:55:51,580 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:51,580 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:55:51,580 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:51,580 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:51,580 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:51,580 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:55:51,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:51,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:51,680 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:55:51,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:55:51,682 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:55:51,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:55:51,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:55:51,683 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:55:51,683 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:51,683 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:55:51,683 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:55:51,684 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:51,684 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:51,685 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:51,685 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:51,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:51,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:51,685 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:51,685 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:55:51,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:51,686 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:51,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:55:51,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:51,686 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:55:51,686 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:55:51,686 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:55:51,686 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:55:51,686 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:51,686 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:55:51,686 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:55:51,686 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:55:51,686 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:51,686 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. started... 2023-03-21 11:55:51,687 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1baade11f622687e2c1237b8441afd49 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-03-21 11:55:51,687 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1baade11f622687e2c1237b8441afd49, store=A 2023-03-21 11:55:51,687 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:55:51,687 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1baade11f622687e2c1237b8441afd49, store=B 2023-03-21 11:55:51,687 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:55:51,687 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1baade11f622687e2c1237b8441afd49, store=C 2023-03-21 11:55:51,687 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:55:51,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(9158): Flush requested on 1baade11f622687e2c1237b8441afd49 2023-03-21 11:55:51,737 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. as already flushing 2023-03-21 11:55:51,753 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:51,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 263 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54864 deadline: 1679399811751, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:51,757 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:51,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 254 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54892 deadline: 1679399811753, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:51,758 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:51,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 262 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54894 deadline: 1679399811754, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:51,857 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:51,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 265 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54864 deadline: 1679399811855, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:51,859 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:51,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 256 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54892 deadline: 1679399811858, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:51,860 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:51,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 264 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54894 deadline: 1679399811858, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:51,985 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:51,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54904 deadline: 1679399811984, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:51,986 DEBUG [Thread-149] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8168 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49., hostname=jenkins-hbase20.apache.org,34369,1679399732030, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-21 11:55:52,010 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:52,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54840 deadline: 1679399812006, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:52,012 DEBUG [Thread-143] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8195 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49., hostname=jenkins-hbase20.apache.org,34369,1679399732030, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-21 11:55:52,062 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:52,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 267 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54864 deadline: 1679399812059, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:52,062 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:52,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 258 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54892 deadline: 1679399812060, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:52,064 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:52,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 266 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54894 deadline: 1679399812062, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:52,095 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=919 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/A/e1105cf8e4b8435695ab118ee62c63a2 2023-03-21 11:55:52,366 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:52,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 269 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54864 deadline: 1679399812363, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:52,367 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:52,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 260 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54892 deadline: 1679399812364, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:52,372 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:52,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 268 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54894 deadline: 1679399812370, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:52,512 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=919 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/B/c2384b6bab7d4c9f970557bc990d65ce 2023-03-21 11:55:52,528 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=919 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/C/77e35199ff9b4a9a8c87ed3753716393 2023-03-21 11:55:52,535 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/A/e1105cf8e4b8435695ab118ee62c63a2 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/e1105cf8e4b8435695ab118ee62c63a2 2023-03-21 11:55:52,544 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/e1105cf8e4b8435695ab118ee62c63a2, entries=150, sequenceid=919, filesize=11.9 K 2023-03-21 11:55:52,546 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/B/c2384b6bab7d4c9f970557bc990d65ce as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/c2384b6bab7d4c9f970557bc990d65ce 2023-03-21 11:55:52,553 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/c2384b6bab7d4c9f970557bc990d65ce, entries=150, sequenceid=919, filesize=11.9 K 2023-03-21 11:55:52,555 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/C/77e35199ff9b4a9a8c87ed3753716393 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/77e35199ff9b4a9a8c87ed3753716393 2023-03-21 11:55:52,560 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/77e35199ff9b4a9a8c87ed3753716393, entries=150, sequenceid=919, filesize=11.9 K 2023-03-21 11:55:52,560 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=87.22 KB/89310 for 1baade11f622687e2c1237b8441afd49 in 873ms, sequenceid=919, compaction requested=true 2023-03-21 11:55:52,560 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1baade11f622687e2c1237b8441afd49: 2023-03-21 11:55:52,561 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:52,561 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:55:52,561 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:55:52,561 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:52,561 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:55:52,561 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:55:52,562 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:52,562 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:52,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:52,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:52,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:52,566 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:52,566 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:55:52,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:52,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:52,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:52,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:52,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:52,568 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:55:52,568 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@77d99e08[Count = 0] remaining members to acquire global barrier 2023-03-21 11:55:52,568 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:55:52,568 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:52,572 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:52,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:52,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:52,572 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:55:52,572 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:55:52,572 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:55:52,573 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:55:52,573 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:52,573 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:55:52,573 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:52,573 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:55:52,573 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:55:52,573 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:55:52,573 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:55:52,574 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:55:52,574 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:55:52,574 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:55:52,575 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:52,575 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:52,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:52,575 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:52,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:52,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:52,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:52,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:52,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:52,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:52,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:52,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:52,576 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:52,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:52,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:52,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:52,577 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:52,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:52,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:52,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:52,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:52,579 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:52,579 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:52,579 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:52,579 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:52,579 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:52,579 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:52,579 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:55:52,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:52,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:52,579 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:55:52,579 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:55:52,579 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:52,580 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:52,580 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:52,580 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:52,580 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:55:52,580 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:55:52,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:52,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:52,680 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:55:52,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:55:52,682 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:55:52,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:55:52,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:55:52,686 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:55:52,686 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:52,686 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:55:52,686 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:55:52,686 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:52,686 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:52,687 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:52,687 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:52,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:52,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:52,688 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:52,688 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:55:52,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:52,688 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:52,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:55:52,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:52,688 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:55:52,698 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:55:52,698 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:55:52,698 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:55:52,698 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:52,698 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:55:52,710 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:55:52,710 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:52,710 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:55:52,710 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. started... 2023-03-21 11:55:52,710 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1baade11f622687e2c1237b8441afd49 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-03-21 11:55:52,710 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1baade11f622687e2c1237b8441afd49, store=A 2023-03-21 11:55:52,711 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:55:52,711 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1baade11f622687e2c1237b8441afd49, store=B 2023-03-21 11:55:52,711 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:55:52,711 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1baade11f622687e2c1237b8441afd49, store=C 2023-03-21 11:55:52,711 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:55:52,779 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=935 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/A/fbe1f9e6e4cd4d9fa6c24b9ffa49c223 2023-03-21 11:55:52,856 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=935 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/B/9d5003fc12d1450ba50020cd62f33df1 2023-03-21 11:55:52,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(9158): Flush requested on 1baade11f622687e2c1237b8441afd49 2023-03-21 11:55:52,872 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. as already flushing 2023-03-21 11:55:52,899 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:52,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 273 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54894 deadline: 1679399812895, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:52,904 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=935 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/C/56a5d0aae2b04c5fb9fae7138cc79169 2023-03-21 11:55:52,906 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:52,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 268 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54892 deadline: 1679399812900, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:52,906 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:52,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 276 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54864 deadline: 1679399812900, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:52,914 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/A/fbe1f9e6e4cd4d9fa6c24b9ffa49c223 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/fbe1f9e6e4cd4d9fa6c24b9ffa49c223 2023-03-21 11:55:52,920 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/fbe1f9e6e4cd4d9fa6c24b9ffa49c223, entries=150, sequenceid=935, filesize=11.9 K 2023-03-21 11:55:52,922 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/B/9d5003fc12d1450ba50020cd62f33df1 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/9d5003fc12d1450ba50020cd62f33df1 2023-03-21 11:55:52,931 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/9d5003fc12d1450ba50020cd62f33df1, entries=150, sequenceid=935, filesize=11.9 K 2023-03-21 11:55:52,933 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/C/56a5d0aae2b04c5fb9fae7138cc79169 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/56a5d0aae2b04c5fb9fae7138cc79169 2023-03-21 11:55:52,945 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/56a5d0aae2b04c5fb9fae7138cc79169, entries=150, sequenceid=935, filesize=11.9 K 2023-03-21 11:55:52,950 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=93.93 KB/96180 for 1baade11f622687e2c1237b8441afd49 in 240ms, sequenceid=935, compaction requested=true 2023-03-21 11:55:52,950 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1baade11f622687e2c1237b8441afd49: 2023-03-21 11:55:52,950 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:52,950 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:55:52,950 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:55:52,950 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:52,951 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:55:52,951 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:55:52,952 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:52,952 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:52,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:52,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:52,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:52,952 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:52,952 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:55:52,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:52,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:52,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:52,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:52,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:52,954 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:55:52,954 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2afe8ed2[Count = 0] remaining members to acquire global barrier 2023-03-21 11:55:52,954 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:55:52,954 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:52,955 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:52,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:52,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:52,955 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:55:52,956 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:55:52,956 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:55:52,956 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:52,956 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:55:52,958 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:55:52,958 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:52,958 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:55:52,958 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:52,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:52,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:52,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:52,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:52,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:52,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:52,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:52,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:52,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:52,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:52,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:55:52,961 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:55:52,961 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:55:52,961 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:55:52,961 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:55:52,961 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:55:52,961 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:55:52,964 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:52,964 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:52,964 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:52,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:52,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:52,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:52,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:52,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:52,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:52,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:52,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:52,965 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:52,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:52,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:52,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:52,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:52,966 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:52,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:52,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:52,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:52,968 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:52,968 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:52,968 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:52,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:52,968 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:52,968 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:55:52,968 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:55:52,968 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:55:52,968 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:52,969 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:55:52,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:52,968 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:52,969 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:55:52,969 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:52,969 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:52,969 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:52,969 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:52,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:52,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:53,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(9158): Flush requested on 1baade11f622687e2c1237b8441afd49 2023-03-21 11:55:53,004 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1baade11f622687e2c1237b8441afd49 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-03-21 11:55:53,004 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1baade11f622687e2c1237b8441afd49, store=A 2023-03-21 11:55:53,004 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:55:53,004 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1baade11f622687e2c1237b8441afd49, store=B 2023-03-21 11:55:53,004 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:55:53,005 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1baade11f622687e2c1237b8441afd49, store=C 2023-03-21 11:55:53,005 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:55:53,015 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=953 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/A/2aa8b2fc68d54ec4846aa1139bb20595 2023-03-21 11:55:53,026 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:53,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 281 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54864 deadline: 1679399813022, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:53,026 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:53,027 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=953 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/B/37d83a86b3f7491eb22a9364f1ad4b35 2023-03-21 11:55:53,027 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:53,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 280 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54894 deadline: 1679399813022, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:53,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 274 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54892 deadline: 1679399813023, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:53,069 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:55:53,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:55:53,071 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:55:53,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:55:53,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:55:53,072 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:55:53,072 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:53,072 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:55:53,072 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:55:53,072 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:53,072 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:53,073 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:53,073 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:53,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:53,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:53,073 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:53,073 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:55:53,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:53,074 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:53,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:55:53,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:53,074 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:55:53,074 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:55:53,074 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:55:53,074 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:55:53,075 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:53,075 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:55:53,075 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:55:53,075 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:55:53,075 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:53,075 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. started... 2023-03-21 11:55:53,075 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. as already flushing 2023-03-21 11:55:53,075 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:53,075 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:55:53,075 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:55:53,075 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:53,075 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:55:53,075 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:55:53,076 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:53,076 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:53,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:53,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:53,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:53,076 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:53,076 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:55:53,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:53,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:53,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:53,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:53,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:53,077 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:55:53,078 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7e5606c5[Count = 0] remaining members to acquire global barrier 2023-03-21 11:55:53,078 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:55:53,078 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:53,078 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:53,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:53,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:53,078 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:55:53,078 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:55:53,078 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:55:53,078 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:53,078 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:55:53,079 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:53,079 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:55:53,079 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:53,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:53,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:53,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:53,079 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:55:53,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:53,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:53,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:53,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:53,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:53,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:53,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:53,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:55:53,081 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:55:53,081 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:55:53,081 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:55:53,081 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:55:53,082 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:55:53,082 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:55:53,082 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:53,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:53,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:53,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:53,083 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:53,083 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:53,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:53,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:53,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:53,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:53,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:53,084 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:53,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:53,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:53,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:53,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:53,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:53,085 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:53,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:53,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:53,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:53,087 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:53,087 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:53,087 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:53,087 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:53,087 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:55:53,087 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:55:53,087 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:53,087 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:53,087 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:53,087 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:53,087 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:53,087 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:53,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:53,087 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:55:53,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:53,088 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:55:53,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:53,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:53,129 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:53,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 283 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54864 deadline: 1679399813127, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:53,129 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:53,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 282 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54894 deadline: 1679399813129, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:53,129 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:53,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 276 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54892 deadline: 1679399813129, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:53,188 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:55:53,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:55:53,189 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:55:53,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:55:53,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:55:53,191 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:55:53,191 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:53,191 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:55:53,191 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:55:53,191 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:53,191 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:53,192 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:53,192 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:53,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:53,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:53,192 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:53,192 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:55:53,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:53,192 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:53,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:55:53,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:53,193 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:55:53,193 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:55:53,193 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:55:53,193 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:55:53,193 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:53,193 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:55:53,193 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:55:53,193 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:55:53,193 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:53,193 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. started... 2023-03-21 11:55:53,193 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. as already flushing 2023-03-21 11:55:53,194 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:53,194 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:55:53,194 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:55:53,194 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:53,194 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:55:53,194 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:55:53,195 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:53,195 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:53,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:53,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:53,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:53,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:53,195 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:53,195 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:55:53,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:53,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:53,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:53,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:53,196 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:55:53,196 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:55:53,196 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c90db45[Count = 0] remaining members to acquire global barrier 2023-03-21 11:55:53,196 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:53,197 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:53,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:53,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:53,197 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:55:53,197 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:53,197 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:55:53,197 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:55:53,197 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:55:53,199 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:53,199 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:55:53,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:53,199 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:53,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:53,199 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:55:53,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:53,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:53,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:53,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:53,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:53,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:53,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:53,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:53,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:55:53,201 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:55:53,201 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:55:53,201 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:55:53,202 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:55:53,202 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:55:53,202 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:55:53,203 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:53,203 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:53,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:53,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:53,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:53,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:53,203 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:53,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:53,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:53,203 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:53,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:53,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:53,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:53,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:53,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:53,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:53,204 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:53,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:53,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:53,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:53,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:53,205 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:53,205 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:53,205 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:53,205 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:55:53,205 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:55:53,205 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:53,205 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:53,206 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:53,206 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:53,206 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:53,205 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:53,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:53,206 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:53,206 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:55:53,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:53,206 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:55:53,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:53,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:53,206 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:55:53,306 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:55:53,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:55:53,308 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:55:53,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:55:53,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:55:53,310 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:55:53,310 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:53,310 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:55:53,310 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:55:53,310 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:53,310 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:53,311 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:53,311 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:53,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:53,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:53,311 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:53,311 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:55:53,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:53,312 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:53,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:55:53,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:53,312 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:55:53,312 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:55:53,312 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:55:53,312 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:55:53,312 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:53,312 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:55:53,312 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:55:53,312 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:55:53,312 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:53,313 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. started... 2023-03-21 11:55:53,313 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. as already flushing 2023-03-21 11:55:53,313 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:53,313 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:55:53,313 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:55:53,313 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:53,313 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:55:53,313 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:55:53,314 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:53,314 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:53,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:53,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:53,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:53,315 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:53,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:53,315 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:55:53,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:53,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:53,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:53,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:53,316 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:55:53,316 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@27823f01[Count = 0] remaining members to acquire global barrier 2023-03-21 11:55:53,316 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:55:53,316 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:53,317 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:53,317 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:55:53,317 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:53,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:53,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:53,317 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:55:53,317 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:55:53,317 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:55:53,318 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:53,318 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:55:53,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:53,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:53,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:53,318 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:53,318 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:55:53,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:53,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:53,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:53,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:53,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:53,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:53,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:53,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:55:53,320 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:55:53,320 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:55:53,320 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:55:53,320 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:55:53,320 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:55:53,320 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:55:53,321 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:53,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:53,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:53,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:53,321 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:53,322 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:53,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:53,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:53,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:53,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:53,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:53,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:53,322 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:53,323 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:53,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:53,325 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:53,325 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:53,325 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:53,326 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:53,326 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:53,326 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:55:53,326 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:53,326 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:53,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:53,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:53,326 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:53,326 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:55:53,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:53,326 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:53,326 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:53,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:53,326 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:55:53,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:53,326 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:55:53,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:53,327 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:55:53,334 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:53,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 284 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54894 deadline: 1679399813330, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:53,334 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:53,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 278 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54892 deadline: 1679399813330, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:53,334 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:53,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 285 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54864 deadline: 1679399813331, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:53,426 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:55:53,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:55:53,429 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:55:53,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:55:53,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:55:53,433 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:55:53,433 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:53,433 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:55:53,433 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:55:53,433 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:53,434 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:53,434 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:53,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:53,434 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:53,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:53,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:53,435 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:53,435 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:55:53,435 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:53,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:55:53,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:53,435 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:55:53,436 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:55:53,436 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:55:53,436 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:55:53,436 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:53,436 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:55:53,436 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:55:53,436 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:55:53,436 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:53,436 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. started... 2023-03-21 11:55:53,436 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. as already flushing 2023-03-21 11:55:53,436 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:53,436 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:55:53,437 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:55:53,437 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:53,437 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:55:53,437 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:55:53,438 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:53,438 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:53,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:53,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:53,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:53,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:53,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:53,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:53,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:53,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:53,439 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:55:53,439 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7954bd3d[Count = 0] remaining members to acquire global barrier 2023-03-21 11:55:53,439 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:55:53,440 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:53,440 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=953 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/C/fbdbac0fa4d24763b5e55625ff3a5b4a 2023-03-21 11:55:53,441 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:53,441 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:55:53,442 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:53,442 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:53,442 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:55:53,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:53,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:53,443 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:55:53,443 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:55:53,443 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:55:53,444 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:53,444 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:55:53,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:53,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:53,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:53,444 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:53,444 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:55:53,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:53,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:53,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:53,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:53,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:53,446 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/A/2aa8b2fc68d54ec4846aa1139bb20595 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/2aa8b2fc68d54ec4846aa1139bb20595 2023-03-21 11:55:53,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:53,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:53,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:55:53,448 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:55:53,448 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:55:53,448 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:55:53,448 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:55:53,448 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:55:53,448 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:55:53,449 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:53,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:53,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:53,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:53,449 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:53,449 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:53,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:53,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:53,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:53,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:53,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:53,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:53,450 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:53,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:53,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:53,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:53,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:53,452 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:53,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:53,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:53,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:53,455 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:53,455 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:53,455 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:53,455 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:53,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:53,455 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:53,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:53,455 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:53,455 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:53,455 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:53,455 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:53,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:53,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:53,456 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:53,456 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:55:53,456 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:55:53,456 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:55:53,456 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:55:53,459 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/2aa8b2fc68d54ec4846aa1139bb20595, entries=200, sequenceid=953, filesize=14.3 K 2023-03-21 11:55:53,463 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/B/37d83a86b3f7491eb22a9364f1ad4b35 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/37d83a86b3f7491eb22a9364f1ad4b35 2023-03-21 11:55:53,469 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/37d83a86b3f7491eb22a9364f1ad4b35, entries=150, sequenceid=953, filesize=11.9 K 2023-03-21 11:55:53,470 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/C/fbdbac0fa4d24763b5e55625ff3a5b4a as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/fbdbac0fa4d24763b5e55625ff3a5b4a 2023-03-21 11:55:53,488 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/fbdbac0fa4d24763b5e55625ff3a5b4a, entries=150, sequenceid=953, filesize=11.9 K 2023-03-21 11:55:53,489 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 1baade11f622687e2c1237b8441afd49 in 485ms, sequenceid=953, compaction requested=true 2023-03-21 11:55:53,490 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1baade11f622687e2c1237b8441afd49: 2023-03-21 11:55:53,490 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-21 11:55:53,490 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-21 11:55:53,490 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-21 11:55:53,490 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-21 11:55:53,490 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-21 11:55:53,492 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 73662 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-21 11:55:53,492 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 76102 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-21 11:55:53,492 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1912): 1baade11f622687e2c1237b8441afd49/B is initiating minor compaction (all files) 2023-03-21 11:55:53,492 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1912): 1baade11f622687e2c1237b8441afd49/A is initiating minor compaction (all files) 2023-03-21 11:55:53,492 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1baade11f622687e2c1237b8441afd49/B in TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:53,492 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 1baade11f622687e2c1237b8441afd49/A in TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:53,492 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/33cf0b3361084c078149f0e649518554, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/cbc30cb2011d42d78d763b7fcb2fcba6, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/00f15a59a24247b8b61cd5360a185ee8, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/e1105cf8e4b8435695ab118ee62c63a2, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/fbe1f9e6e4cd4d9fa6c24b9ffa49c223, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/2aa8b2fc68d54ec4846aa1139bb20595] into tmpdir=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp, totalSize=74.3 K 2023-03-21 11:55:53,492 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/9653f09670654169b0bb33dc6a2431a5, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/b7286c70fb974fee938b24e92badf461, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/9b1dfc4ceaf7414aba148680881a7954, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/c2384b6bab7d4c9f970557bc990d65ce, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/9d5003fc12d1450ba50020cd62f33df1, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/37d83a86b3f7491eb22a9364f1ad4b35] into tmpdir=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp, totalSize=71.9 K 2023-03-21 11:55:53,493 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 33cf0b3361084c078149f0e649518554, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=867, earliestPutTs=1679399747444 2023-03-21 11:55:53,493 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 9653f09670654169b0bb33dc6a2431a5, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=867, earliestPutTs=1679399747444 2023-03-21 11:55:53,493 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting cbc30cb2011d42d78d763b7fcb2fcba6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=884, earliestPutTs=1679399747571 2023-03-21 11:55:53,493 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting b7286c70fb974fee938b24e92badf461, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=884, earliestPutTs=1679399747571 2023-03-21 11:55:53,493 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 00f15a59a24247b8b61cd5360a185ee8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=902, earliestPutTs=1679399749483 2023-03-21 11:55:53,493 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 9b1dfc4ceaf7414aba148680881a7954, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=902, earliestPutTs=1679399749483 2023-03-21 11:55:53,493 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting e1105cf8e4b8435695ab118ee62c63a2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=919, earliestPutTs=1679399750623 2023-03-21 11:55:53,494 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting c2384b6bab7d4c9f970557bc990d65ce, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=919, earliestPutTs=1679399750623 2023-03-21 11:55:53,494 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting fbe1f9e6e4cd4d9fa6c24b9ffa49c223, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=935, earliestPutTs=1679399751750 2023-03-21 11:55:53,494 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 9d5003fc12d1450ba50020cd62f33df1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=935, earliestPutTs=1679399751750 2023-03-21 11:55:53,494 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 37d83a86b3f7491eb22a9364f1ad4b35, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=953, earliestPutTs=1679399752889 2023-03-21 11:55:53,495 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 2aa8b2fc68d54ec4846aa1139bb20595, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=953, earliestPutTs=1679399752889 2023-03-21 11:55:53,511 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] throttle.PressureAwareThroughputController(145): 1baade11f622687e2c1237b8441afd49#B#compaction#66 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-21 11:55:53,522 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] throttle.PressureAwareThroughputController(145): 1baade11f622687e2c1237b8441afd49#A#compaction#67 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-21 11:55:53,556 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:55:53,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:55:53,559 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:55:53,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:55:53,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:55:53,564 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:55:53,564 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:53,564 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:55:53,564 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:55:53,564 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:53,564 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:53,565 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:53,565 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:53,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:53,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:53,566 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:53,566 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:55:53,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:53,566 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:53,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:55:53,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:53,567 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:55:53,567 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:55:53,567 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:55:53,567 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:55:53,567 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:53,567 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:55:53,574 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:55:53,574 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:55:53,574 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:53,574 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. started... 2023-03-21 11:55:53,574 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1baade11f622687e2c1237b8441afd49 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-03-21 11:55:53,574 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1baade11f622687e2c1237b8441afd49, store=A 2023-03-21 11:55:53,574 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:55:53,575 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1baade11f622687e2c1237b8441afd49, store=B 2023-03-21 11:55:53,575 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:55:53,575 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1baade11f622687e2c1237b8441afd49, store=C 2023-03-21 11:55:53,575 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:55:53,604 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=967 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/A/2de7da9ae5314538a7fe459110acc268 2023-03-21 11:55:53,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(9158): Flush requested on 1baade11f622687e2c1237b8441afd49 2023-03-21 11:55:53,639 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. as already flushing 2023-03-21 11:55:53,656 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:53,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 291 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54894 deadline: 1679399813654, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:53,656 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:53,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 292 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54864 deadline: 1679399813654, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:53,658 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:53,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 285 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54892 deadline: 1679399813655, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:53,764 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:53,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 293 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54894 deadline: 1679399813757, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:53,764 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:53,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 294 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54864 deadline: 1679399813759, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:53,765 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:53,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 287 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54892 deadline: 1679399813760, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:53,960 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/B/5edf36bacd6d4c51908da851b0d463a2 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/5edf36bacd6d4c51908da851b0d463a2 2023-03-21 11:55:53,968 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 1baade11f622687e2c1237b8441afd49/B of 1baade11f622687e2c1237b8441afd49 into 5edf36bacd6d4c51908da851b0d463a2(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-21 11:55:53,968 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1baade11f622687e2c1237b8441afd49: 2023-03-21 11:55:53,968 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49., storeName=1baade11f622687e2c1237b8441afd49/B, priority=10, startTime=1679399753490; duration=0sec 2023-03-21 11:55:53,969 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-21 11:55:53,969 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-21 11:55:53,970 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:53,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 296 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54864 deadline: 1679399813967, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:53,972 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:53,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 289 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54892 deadline: 1679399813967, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:53,972 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:53,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 295 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54894 deadline: 1679399813970, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:53,978 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 73662 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-21 11:55:53,981 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1912): 1baade11f622687e2c1237b8441afd49/C is initiating minor compaction (all files) 2023-03-21 11:55:53,981 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1baade11f622687e2c1237b8441afd49/C in TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:53,981 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/7f32ec67ab3f4ec8afadfc498bc63068, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/f9de91cdac0944069e36c00cb76e0a3b, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/5193289533354700ada1dddf509941dd, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/77e35199ff9b4a9a8c87ed3753716393, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/56a5d0aae2b04c5fb9fae7138cc79169, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/fbdbac0fa4d24763b5e55625ff3a5b4a] into tmpdir=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp, totalSize=71.9 K 2023-03-21 11:55:53,982 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 7f32ec67ab3f4ec8afadfc498bc63068, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=867, earliestPutTs=1679399747444 2023-03-21 11:55:53,982 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting f9de91cdac0944069e36c00cb76e0a3b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=884, earliestPutTs=1679399747571 2023-03-21 11:55:53,983 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 5193289533354700ada1dddf509941dd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=902, earliestPutTs=1679399749483 2023-03-21 11:55:53,984 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 77e35199ff9b4a9a8c87ed3753716393, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=919, earliestPutTs=1679399750623 2023-03-21 11:55:53,984 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 56a5d0aae2b04c5fb9fae7138cc79169, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=935, earliestPutTs=1679399751750 2023-03-21 11:55:53,985 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting fbdbac0fa4d24763b5e55625ff3a5b4a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=953, earliestPutTs=1679399752889 2023-03-21 11:55:53,985 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/A/8214a0171e384bf7875c252e1c55cb53 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/8214a0171e384bf7875c252e1c55cb53 2023-03-21 11:55:53,996 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 1baade11f622687e2c1237b8441afd49/A of 1baade11f622687e2c1237b8441afd49 into 8214a0171e384bf7875c252e1c55cb53(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-21 11:55:53,996 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1baade11f622687e2c1237b8441afd49: 2023-03-21 11:55:53,996 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49., storeName=1baade11f622687e2c1237b8441afd49/A, priority=10, startTime=1679399753490; duration=0sec 2023-03-21 11:55:53,996 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-21 11:55:54,016 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] throttle.PressureAwareThroughputController(145): 1baade11f622687e2c1237b8441afd49#C#compaction#70 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-21 11:55:54,019 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=967 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/B/78c678df2e484924a1c5baca85548bd8 2023-03-21 11:55:54,047 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=967 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/C/b9382c6a30d740e1817cf96c59500eea 2023-03-21 11:55:54,052 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/A/2de7da9ae5314538a7fe459110acc268 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/2de7da9ae5314538a7fe459110acc268 2023-03-21 11:55:54,057 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/2de7da9ae5314538a7fe459110acc268, entries=150, sequenceid=967, filesize=11.9 K 2023-03-21 11:55:54,058 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/B/78c678df2e484924a1c5baca85548bd8 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/78c678df2e484924a1c5baca85548bd8 2023-03-21 11:55:54,064 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/78c678df2e484924a1c5baca85548bd8, entries=150, sequenceid=967, filesize=11.9 K 2023-03-21 11:55:54,066 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/C/b9382c6a30d740e1817cf96c59500eea as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/b9382c6a30d740e1817cf96c59500eea 2023-03-21 11:55:54,072 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/b9382c6a30d740e1817cf96c59500eea, entries=150, sequenceid=967, filesize=11.9 K 2023-03-21 11:55:54,073 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 1baade11f622687e2c1237b8441afd49 in 499ms, sequenceid=967, compaction requested=false 2023-03-21 11:55:54,073 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1baade11f622687e2c1237b8441afd49: 2023-03-21 11:55:54,073 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:54,073 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:55:54,073 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:55:54,073 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:54,073 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:55:54,073 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:55:54,074 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:54,074 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:54,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:54,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:54,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:54,077 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:54,077 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:55:54,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:54,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:54,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:54,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:54,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:54,083 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:55:54,083 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@37faecfc[Count = 0] remaining members to acquire global barrier 2023-03-21 11:55:54,083 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:55:54,083 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:54,088 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:54,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:54,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:54,088 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:55:54,088 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:55:54,088 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:55:54,088 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:54,088 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:55:54,089 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:55:54,089 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:54,089 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:54,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:54,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:54,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:54,089 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:55:54,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:54,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:54,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:54,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:54,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:54,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:54,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:54,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:55:54,092 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:55:54,092 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:55:54,092 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:55:54,092 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:55:54,092 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:55:54,092 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:55:54,093 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:54,093 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:54,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:54,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:54,093 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:54,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:54,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:54,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:54,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:54,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:54,094 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:54,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:54,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:54,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:54,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:54,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:54,095 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:54,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:54,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:54,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:54,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:54,097 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:54,097 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:54,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:54,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:54,097 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:54,097 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:55:54,097 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:55:54,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:54,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:54,098 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:55:54,098 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:54,098 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:54,098 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:54,098 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:54,098 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:54,098 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:54,098 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:54,101 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:55:54,101 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:55:54,201 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:55:54,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:55:54,203 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:55:54,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:55:54,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:55:54,204 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:55:54,204 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:54,204 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:55:54,204 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:55:54,204 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:54,204 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:54,205 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:54,205 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:54,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:54,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:54,205 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:54,205 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:55:54,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:54,205 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:54,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:55:54,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:54,206 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:55:54,206 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:55:54,206 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:55:54,206 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:55:54,206 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:54,206 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:55:54,206 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:55:54,206 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:55:54,206 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:54,206 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. started... 2023-03-21 11:55:54,207 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1baade11f622687e2c1237b8441afd49 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-03-21 11:55:54,207 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1baade11f622687e2c1237b8441afd49, store=A 2023-03-21 11:55:54,207 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:55:54,207 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1baade11f622687e2c1237b8441afd49, store=B 2023-03-21 11:55:54,207 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:55:54,207 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1baade11f622687e2c1237b8441afd49, store=C 2023-03-21 11:55:54,207 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:55:54,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(9158): Flush requested on 1baade11f622687e2c1237b8441afd49 2023-03-21 11:55:54,275 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. as already flushing 2023-03-21 11:55:54,286 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:54,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 301 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54864 deadline: 1679399814285, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:54,287 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:54,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 295 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54892 deadline: 1679399814285, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:54,287 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:54,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 301 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54894 deadline: 1679399814286, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:54,389 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:54,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 303 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54864 deadline: 1679399814387, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:54,390 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:54,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 297 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54892 deadline: 1679399814388, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:54,392 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:54,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 303 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54894 deadline: 1679399814388, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:54,441 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/C/a2dc6bb92bac481c8458e6d8d016f6a1 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/a2dc6bb92bac481c8458e6d8d016f6a1 2023-03-21 11:55:54,447 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 1baade11f622687e2c1237b8441afd49/C of 1baade11f622687e2c1237b8441afd49 into a2dc6bb92bac481c8458e6d8d016f6a1(size=12.6 K), total size for store is 24.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-21 11:55:54,447 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1baade11f622687e2c1237b8441afd49: 2023-03-21 11:55:54,447 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49., storeName=1baade11f622687e2c1237b8441afd49/C, priority=10, startTime=1679399753490; duration=0sec 2023-03-21 11:55:54,447 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-21 11:55:54,593 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:54,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 305 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54864 deadline: 1679399814592, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:54,593 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:54,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 299 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54892 deadline: 1679399814592, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:54,594 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:54,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 305 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54894 deadline: 1679399814592, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:54,615 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=987 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/A/9aa37aa540cd47f4bdbcc8607923a1a3 2023-03-21 11:55:54,896 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:54,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 307 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54864 deadline: 1679399814895, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:54,896 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:54,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 307 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54894 deadline: 1679399814895, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:54,897 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:54,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 301 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54892 deadline: 1679399814896, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:55,027 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=987 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/B/895440721ca54307a6567a8ece1dd90d 2023-03-21 11:55:55,038 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=987 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/C/0aed0c6743ac423abcc70c821d9f2035 2023-03-21 11:55:55,043 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/A/9aa37aa540cd47f4bdbcc8607923a1a3 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/9aa37aa540cd47f4bdbcc8607923a1a3 2023-03-21 11:55:55,054 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/9aa37aa540cd47f4bdbcc8607923a1a3, entries=150, sequenceid=987, filesize=11.9 K 2023-03-21 11:55:55,059 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/B/895440721ca54307a6567a8ece1dd90d as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/895440721ca54307a6567a8ece1dd90d 2023-03-21 11:55:55,064 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/895440721ca54307a6567a8ece1dd90d, entries=150, sequenceid=987, filesize=11.9 K 2023-03-21 11:55:55,065 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/C/0aed0c6743ac423abcc70c821d9f2035 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/0aed0c6743ac423abcc70c821d9f2035 2023-03-21 11:55:55,072 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/0aed0c6743ac423abcc70c821d9f2035, entries=150, sequenceid=987, filesize=11.9 K 2023-03-21 11:55:55,073 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 1baade11f622687e2c1237b8441afd49 in 867ms, sequenceid=987, compaction requested=true 2023-03-21 11:55:55,073 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1baade11f622687e2c1237b8441afd49: 2023-03-21 11:55:55,073 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:55,073 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:55:55,073 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:55:55,073 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:55,073 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:55:55,073 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:55:55,078 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:55,079 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:55,079 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:55:55,080 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:55,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:55,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:55,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:55,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:55,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:55,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:55,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:55,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:55,083 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:55:55,083 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2daa53b5[Count = 0] remaining members to acquire global barrier 2023-03-21 11:55:55,083 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:55:55,083 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:55,097 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:55,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:55,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:55,098 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:55:55,098 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:55:55,098 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:55:55,098 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:55,098 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:55:55,104 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:55:55,104 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:55,104 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:55,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:55,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:55,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:55,104 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:55:55,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:55,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:55,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:55,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:55,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:55,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:55,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:55,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:55:55,107 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:55:55,107 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:55:55,107 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:55:55,108 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:55:55,108 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:55:55,108 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:55:55,114 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:55,114 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:55,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:55,114 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:55,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:55,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:55,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:55,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:55,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:55,115 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:55,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:55,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:55,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:55,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:55,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:55,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:55,117 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:55,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:55,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:55,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:55,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:55,124 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:55,124 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:55,124 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:55,124 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:55,124 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:55,124 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:55,124 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:55,124 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:55,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:55,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:55,124 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:55,124 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:55,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:55,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:55,128 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:55:55,128 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:55:55,129 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:55:55,129 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:55:55,129 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:55:55,229 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:55:55,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:55:55,231 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:55:55,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:55:55,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:55:55,231 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:55:55,231 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:55,231 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:55:55,231 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:55:55,232 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:55,232 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:55,232 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:55,232 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:55,232 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:55,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:55,233 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:55,233 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:55:55,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:55,233 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:55,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:55:55,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:55,234 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:55:55,234 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:55:55,234 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:55:55,234 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:55:55,234 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:55,234 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:55:55,238 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:55:55,238 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:55:55,238 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:55,238 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. started... 2023-03-21 11:55:55,238 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1baade11f622687e2c1237b8441afd49 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-03-21 11:55:55,238 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1baade11f622687e2c1237b8441afd49, store=A 2023-03-21 11:55:55,238 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:55:55,239 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1baade11f622687e2c1237b8441afd49, store=B 2023-03-21 11:55:55,239 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:55:55,239 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1baade11f622687e2c1237b8441afd49, store=C 2023-03-21 11:55:55,239 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:55:55,250 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=1002 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/A/b4f70c225a65499c8e4cc23f04c9e3ac 2023-03-21 11:55:55,266 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=1002 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/B/f58907ca78ee499ca9f4804151859caa 2023-03-21 11:55:55,276 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=1002 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/C/eeb55826ed4c4c6dbb5f6bd990fc7bd8 2023-03-21 11:55:55,282 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/A/b4f70c225a65499c8e4cc23f04c9e3ac as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/b4f70c225a65499c8e4cc23f04c9e3ac 2023-03-21 11:55:55,288 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/b4f70c225a65499c8e4cc23f04c9e3ac, entries=150, sequenceid=1002, filesize=11.9 K 2023-03-21 11:55:55,290 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/B/f58907ca78ee499ca9f4804151859caa as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/f58907ca78ee499ca9f4804151859caa 2023-03-21 11:55:55,306 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/f58907ca78ee499ca9f4804151859caa, entries=150, sequenceid=1002, filesize=11.9 K 2023-03-21 11:55:55,307 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/C/eeb55826ed4c4c6dbb5f6bd990fc7bd8 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/eeb55826ed4c4c6dbb5f6bd990fc7bd8 2023-03-21 11:55:55,325 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/eeb55826ed4c4c6dbb5f6bd990fc7bd8, entries=150, sequenceid=1002, filesize=11.9 K 2023-03-21 11:55:55,326 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=0 B/0 for 1baade11f622687e2c1237b8441afd49 in 88ms, sequenceid=1002, compaction requested=true 2023-03-21 11:55:55,327 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1baade11f622687e2c1237b8441afd49: 2023-03-21 11:55:55,327 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:55,327 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:55:55,327 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:55:55,327 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:55,327 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:55:55,327 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:55:55,328 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:55,328 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:55,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:55,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:55,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:55,328 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:55,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:55,328 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:55:55,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:55,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:55,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:55,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:55,330 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:55:55,330 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b661b36[Count = 0] remaining members to acquire global barrier 2023-03-21 11:55:55,330 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:55:55,330 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:55,331 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:55,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:55,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:55,331 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:55:55,331 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:55:55,331 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:55:55,331 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:55,331 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:55:55,332 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:55,332 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:55:55,332 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:55,332 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:55:55,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:55,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:55,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:55,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:55,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:55,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:55,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:55,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:55,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:55,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:55,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:55:55,336 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:55:55,336 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:55:55,336 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:55:55,336 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:55:55,336 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:55:55,336 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:55:55,338 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:55,338 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:55,338 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:55,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:55,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:55,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:55,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:55,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:55,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:55,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:55,338 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:55,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:55,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:55,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:55,340 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:55,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:55,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:55,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:55,342 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:55,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:55,343 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:55,343 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:55,343 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:55,343 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:55,343 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:55,343 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:55,343 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:55,343 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:55,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:55,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:55,343 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:55:55,344 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:55:55,344 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:55:55,344 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:55:55,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:55,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:55,344 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:55,344 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:55:55,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(9158): Flush requested on 1baade11f622687e2c1237b8441afd49 2023-03-21 11:55:55,410 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1baade11f622687e2c1237b8441afd49 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-21 11:55:55,410 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1baade11f622687e2c1237b8441afd49, store=A 2023-03-21 11:55:55,410 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:55:55,410 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1baade11f622687e2c1237b8441afd49, store=B 2023-03-21 11:55:55,410 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:55:55,411 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1baade11f622687e2c1237b8441afd49, store=C 2023-03-21 11:55:55,411 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:55:55,426 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=1014 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/A/ac60d45f5967445f9ffec2d6410b9dfb 2023-03-21 11:55:55,435 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:55,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 318 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54864 deadline: 1679399815432, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:55,436 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:55,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 317 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54894 deadline: 1679399815433, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:55,439 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:55,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 312 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54892 deadline: 1679399815436, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:55,444 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:55:55,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:55:55,446 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:55:55,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:55:55,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:55:55,447 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:55:55,447 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:55,447 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:55:55,447 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:55:55,447 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:55,447 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:55,449 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:55,449 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:55,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:55,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:55,449 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:55,449 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:55:55,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:55,449 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:55,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:55:55,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:55,449 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:55:55,450 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:55:55,450 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:55:55,450 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:55:55,450 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:55,450 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:55:55,450 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:55:55,450 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:55:55,450 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:55,450 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. started... 2023-03-21 11:55:55,450 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. as already flushing 2023-03-21 11:55:55,450 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:55,450 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:55:55,450 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:55:55,450 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:55,450 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:55:55,450 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:55:55,451 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:55,451 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:55,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:55,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:55,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:55,451 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:55,451 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:55:55,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:55,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:55,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:55,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:55,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:55,452 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:55:55,452 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:55:55,452 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4787e032[Count = 0] remaining members to acquire global barrier 2023-03-21 11:55:55,452 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:55,453 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:55,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:55,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:55,453 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:55:55,453 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:55:55,453 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:55:55,453 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:55,453 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:55:55,454 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:55:55,454 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:55,454 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:55,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:55,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:55,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:55,454 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:55:55,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:55,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:55,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:55,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:55,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:55,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:55,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:55,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:55:55,456 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:55:55,456 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:55:55,456 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:55:55,456 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:55:55,456 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:55:55,456 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:55:55,456 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:55,456 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:55,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:55,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:55,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:55,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:55,456 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:55,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:55,457 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:55,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:55,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:55,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:55,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:55,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:55,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:55,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:55,458 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:55,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:55,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:55,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:55,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:55,470 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:55,470 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:55,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:55,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:55,471 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:55:55,471 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:55:55,470 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:55,471 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:55:55,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:55,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:55,471 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:55,471 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:55:55,470 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:55,471 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:55:55,471 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:55,471 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:55,471 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:55,471 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:55,471 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:55,539 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:55,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 320 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54864 deadline: 1679399815537, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:55,540 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:55,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 319 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54894 deadline: 1679399815537, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:55,540 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:55,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 314 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54892 deadline: 1679399815540, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:55,571 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:55:55,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:55:55,574 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:55:55,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:55:55,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:55:55,591 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:55:55,591 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:55,591 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:55:55,591 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:55:55,592 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:55,592 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:55,592 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:55,592 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:55,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:55,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:55,593 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:55,593 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:55:55,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:55,593 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:55,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:55:55,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:55,593 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:55:55,594 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:55:55,594 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:55:55,594 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:55:55,594 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:55,594 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:55:55,595 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:55:55,595 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:55:55,595 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:55,595 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. started... 2023-03-21 11:55:55,596 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. as already flushing 2023-03-21 11:55:55,596 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:55,596 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:55:55,596 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:55:55,596 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:55,596 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:55:55,596 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:55:55,597 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:55,597 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:55,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:55,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:55,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:55,597 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:55,597 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:55:55,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:55,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:55,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:55,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:55,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:55,598 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:55:55,598 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:55:55,598 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@27fa6932[Count = 0] remaining members to acquire global barrier 2023-03-21 11:55:55,599 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:55,599 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:55,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:55,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:55,599 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:55:55,599 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:55,599 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:55:55,599 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:55:55,599 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:55:55,600 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:55:55,600 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:55,600 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:55:55,601 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:55,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:55,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:55,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:55,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:55,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:55,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:55,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:55,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:55,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:55,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:55,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:55:55,603 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:55:55,603 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:55:55,603 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:55:55,603 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:55:55,603 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:55:55,603 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:55:55,604 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:55,604 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:55,604 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:55,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:55,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:55,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:55,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:55,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:55,604 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:55,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:55,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:55,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:55,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:55,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:55,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:55,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:55,605 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:55,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:55,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:55,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:55,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:55,607 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:55,607 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:55,607 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:55,607 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:55,607 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:55,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:55,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:55,607 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:55:55,608 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:55:55,607 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:55,607 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:55,607 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:55,608 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:55,608 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:55,608 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:55:55,608 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:55:55,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:55,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:55,608 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:55:55,708 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:55:55,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:55:55,710 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:55:55,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:55:55,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:55:55,712 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:55:55,712 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:55,712 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:55:55,712 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:55:55,712 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:55,712 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:55,715 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:55,715 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:55,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:55,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:55,715 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:55,715 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:55:55,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:55,715 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:55,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:55:55,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:55,716 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:55:55,718 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:55:55,718 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:55:55,718 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:55:55,718 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:55,718 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:55:55,722 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:55:55,722 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:55,722 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. started... 2023-03-21 11:55:55,722 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:55:55,722 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. as already flushing 2023-03-21 11:55:55,722 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:55,722 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:55:55,722 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:55:55,722 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:55,722 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:55:55,722 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:55:55,742 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:55,742 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:55,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:55,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:55,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:55,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:55,743 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:55,743 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:55,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:55,743 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:55:55,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 321 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54894 deadline: 1679399815743, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:55,744 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:55,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 322 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54864 deadline: 1679399815743, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:55,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:55,744 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:55,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 316 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54892 deadline: 1679399815743, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:55,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:55,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:55,745 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:55:55,745 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@58b2a183[Count = 0] remaining members to acquire global barrier 2023-03-21 11:55:55,745 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:55:55,745 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:55,746 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:55,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:55,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:55,746 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:55:55,746 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:55:55,746 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:55:55,747 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:55,747 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:55:55,747 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:55,747 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:55:55,747 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:55,747 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:55:55,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:55,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:55,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:55,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:55,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:55,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:55,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:55,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:55,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:55,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:55,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:55:55,752 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:55:55,752 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:55:55,752 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:55:55,752 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:55:55,752 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:55:55,752 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:55:55,753 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:55,753 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:55,753 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:55,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:55,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:55,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:55,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:55,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:55,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:55,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:55,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:55,754 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:55,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:55,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:55,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:55,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:55,755 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:55,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:55,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:55,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:55,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:55,757 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:55,757 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:55,757 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:55,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:55,758 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:55:55,758 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:55:55,758 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:55,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:55,758 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:55:55,758 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:55,758 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:55,758 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:55,758 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:55,758 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:55:55,758 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:55,758 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:55:55,758 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:55,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:55,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:55,842 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=1014 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/B/3a64dd6349ab466bb4fb6210b7aab131 2023-03-21 11:55:55,858 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:55:55,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:55:55,860 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:55:55,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:55:55,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:55:55,861 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:55:55,861 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:55,861 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:55:55,861 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:55:55,861 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:55,861 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:55,862 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:55,862 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:55,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:55,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:55,862 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:55,862 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:55:55,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:55,863 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:55,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:55:55,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:55,863 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:55:55,863 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:55:55,863 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:55:55,863 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:55:55,863 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:55,863 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:55:55,863 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:55:55,863 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:55:55,863 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:55,863 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. started... 2023-03-21 11:55:55,864 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. as already flushing 2023-03-21 11:55:55,864 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:55,864 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:55:55,864 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:55:55,864 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:55,864 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:55:55,864 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:55:55,865 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:55,865 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:55,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:55,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:55,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:55,865 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:55,865 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:55:55,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:55,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:55,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:55,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:55,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:55,866 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:55:55,866 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@73a6be08[Count = 0] remaining members to acquire global barrier 2023-03-21 11:55:55,866 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:55:55,866 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:55,867 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:55,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:55,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:55,867 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:55,867 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:55:55,867 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:55:55,867 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:55:55,867 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:55:55,868 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:55,868 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:55:55,868 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:55,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:55,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:55,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:55,868 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:55:55,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:55,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:55,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:55,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:55,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:55,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:55,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:55,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:55:55,870 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:55:55,870 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:55:55,870 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:55:55,870 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:55:55,870 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:55:55,870 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:55:55,871 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:55,871 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:55,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:55,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:55,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:55,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:55,872 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:55,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:55,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:55,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:55,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:55,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:55,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:55,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:55,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:55,874 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:55,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:55,875 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:55,876 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:55,876 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:55,876 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:55,876 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:55,876 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:55,876 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:55,876 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:55,876 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:55,876 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:55:55,876 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:55:55,877 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:55,877 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:55,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:55,877 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:55:55,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:55,877 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:55:55,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:55,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:55,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:55,879 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:55:55,977 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:55:55,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:55:55,981 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:55:55,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:55:55,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:55:55,994 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:55:55,994 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:55,994 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:55:55,994 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:55:55,995 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:55,995 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:55,996 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:55,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:55,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:55,996 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:55,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:55,997 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:55,997 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:55:55,997 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:55,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:55:55,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:55,997 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:55:55,998 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:55:55,998 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:55:55,998 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:55:55,998 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:55,998 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:55:56,005 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:55:56,005 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:55:56,005 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:56,006 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. started... 2023-03-21 11:55:56,006 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. as already flushing 2023-03-21 11:55:56,006 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:56,006 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:55:56,006 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:55:56,006 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:56,006 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:55:56,006 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:55:56,007 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:56,007 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:56,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:56,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:56,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:56,008 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:56,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:56,008 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:55:56,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:56,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:56,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:56,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:56,009 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:55:56,009 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@bd8563e[Count = 0] remaining members to acquire global barrier 2023-03-21 11:55:56,009 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:55:56,009 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:56,010 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:56,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:56,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:56,010 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:56,010 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:55:56,010 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:55:56,010 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:55:56,010 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:55:56,011 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:56,011 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:55:56,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:56,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:56,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:56,011 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:56,012 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:55:56,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:56,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:56,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:56,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:56,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:56,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:56,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:56,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:55:56,014 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:55:56,014 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:55:56,014 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:55:56,014 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:55:56,014 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:55:56,014 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:55:56,023 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:56,023 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:56,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:56,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:56,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:56,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:56,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:56,023 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:56,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:56,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:56,023 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:56,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:56,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:56,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:56,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:56,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:56,024 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:56,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:56,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:56,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:56,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:56,029 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:56,029 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:56,029 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:55:56,030 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:55:56,029 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:56,029 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:56,030 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:56,030 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:56,030 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:56,030 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:56,030 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:56,030 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:55:56,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:56,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:56,030 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:55:56,029 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:56,030 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:55:56,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:56,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:56,048 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:56,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 323 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54894 deadline: 1679399816044, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:56,049 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:56,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 324 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54864 deadline: 1679399816046, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:56,049 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:55:56,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 318 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54892 deadline: 1679399816046, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:56,131 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:55:56,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:55:56,132 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:55:56,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:55:56,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:55:56,133 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:55:56,133 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:56,133 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:55:56,133 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:55:56,134 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:56,134 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:56,135 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:56,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:56,135 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:56,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:56,135 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:56,135 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:55:56,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:56,136 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:56,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:55:56,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:56,136 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:55:56,136 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:55:56,136 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:55:56,136 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:55:56,136 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:56,136 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:55:56,137 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:55:56,137 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:55:56,137 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:56,137 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. started... 2023-03-21 11:55:56,137 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. as already flushing 2023-03-21 11:55:56,137 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:56,137 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:55:56,137 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:55:56,137 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:56,137 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:55:56,137 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:55:56,139 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:56,139 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:56,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:56,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:56,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:56,139 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:56,139 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:55:56,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:56,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:56,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:56,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:56,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:56,141 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:55:56,141 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f9f1d8e[Count = 0] remaining members to acquire global barrier 2023-03-21 11:55:56,141 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:55:56,141 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:56,141 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:56,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:56,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:56,141 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:55:56,141 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:55:56,141 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:55:56,142 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:56,142 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:55:56,142 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:56,142 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:55:56,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:56,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:56,142 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:56,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:56,142 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:55:56,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:56,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:56,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:56,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:56,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:56,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:56,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:56,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:55:56,144 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:55:56,144 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:55:56,144 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:55:56,144 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:55:56,144 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:55:56,144 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:55:56,146 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:56,146 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:56,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:56,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:56,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:56,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:56,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:56,146 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:56,146 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:56,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:56,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:56,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:56,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:56,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:56,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:56,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:56,148 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:56,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:56,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:56,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:56,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:56,149 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:56,150 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:56,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:56,150 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:55:56,150 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:55:56,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:56,150 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:56,150 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:56,150 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:56,150 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:56,150 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:56,150 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:56,150 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:56,150 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:56,150 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:55:56,150 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:55:56,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:56,150 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:55:56,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:56,250 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:55:56,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:55:56,252 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:55:56,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:55:56,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:55:56,253 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:55:56,253 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:56,254 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:55:56,254 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:55:56,254 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:56,254 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:56,255 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:56,255 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:56,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:56,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:56,255 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:56,255 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:55:56,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:56,255 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:56,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:55:56,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:56,256 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:55:56,256 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:55:56,256 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:55:56,256 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:55:56,256 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:56,256 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:55:56,257 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:55:56,257 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:55:56,257 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:56,257 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. started... 2023-03-21 11:55:56,257 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. as already flushing 2023-03-21 11:55:56,257 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:56,257 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=1014 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/C/054815745f9844c191e732d28da3d26e 2023-03-21 11:55:56,261 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:55:56,262 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:55:56,262 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:56,262 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:55:56,262 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:55:56,264 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/A/ac60d45f5967445f9ffec2d6410b9dfb as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/ac60d45f5967445f9ffec2d6410b9dfb 2023-03-21 11:55:56,264 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:56,264 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:56,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:56,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:56,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:56,265 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:56,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:56,265 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:55:56,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:56,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:56,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:56,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:56,266 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:55:56,266 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@32ec0547[Count = 0] remaining members to acquire global barrier 2023-03-21 11:55:56,266 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:55:56,266 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:56,267 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:56,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:56,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:56,267 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:55:56,267 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:55:56,267 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:55:56,268 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:56,268 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:55:56,269 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:56,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:56,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:56,269 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:55:56,269 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:56,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:56,269 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:55:56,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:56,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:56,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:56,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:56,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:56,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:56,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:56,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:55:56,271 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:55:56,272 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:55:56,272 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:55:56,272 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:55:56,272 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:55:56,272 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:55:56,281 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:56,281 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:56,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:56,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:56,281 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:56,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:56,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:56,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:56,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:56,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:56,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:56,282 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:56,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:56,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:56,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:56,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:56,283 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:56,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:56,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:56,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:56,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:56,288 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/ac60d45f5967445f9ffec2d6410b9dfb, entries=200, sequenceid=1014, filesize=14.3 K 2023-03-21 11:55:56,289 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/B/3a64dd6349ab466bb4fb6210b7aab131 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/3a64dd6349ab466bb4fb6210b7aab131 2023-03-21 11:55:56,289 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:56,289 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:55:56,290 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:55:56,289 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:56,290 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:56,290 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:56,290 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:56,290 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:56,290 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:56,290 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:55:56,289 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:56,290 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:55:56,290 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:56,290 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:56,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:56,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:56,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:56,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:56,299 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/3a64dd6349ab466bb4fb6210b7aab131, entries=150, sequenceid=1014, filesize=11.9 K 2023-03-21 11:55:56,301 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/C/054815745f9844c191e732d28da3d26e as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/054815745f9844c191e732d28da3d26e 2023-03-21 11:55:56,306 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/054815745f9844c191e732d28da3d26e, entries=150, sequenceid=1014, filesize=11.9 K 2023-03-21 11:55:56,307 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 1baade11f622687e2c1237b8441afd49 in 897ms, sequenceid=1014, compaction requested=true 2023-03-21 11:55:56,308 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1baade11f622687e2c1237b8441afd49: 2023-03-21 11:55:56,308 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-21 11:55:56,308 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-21 11:55:56,308 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-21 11:55:56,308 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-21 11:55:56,308 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-21 11:55:56,310 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 64120 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-21 11:55:56,310 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1912): 1baade11f622687e2c1237b8441afd49/A is initiating minor compaction (all files) 2023-03-21 11:55:56,310 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 1baade11f622687e2c1237b8441afd49/A in TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:56,311 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/8214a0171e384bf7875c252e1c55cb53, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/2de7da9ae5314538a7fe459110acc268, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/9aa37aa540cd47f4bdbcc8607923a1a3, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/b4f70c225a65499c8e4cc23f04c9e3ac, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/ac60d45f5967445f9ffec2d6410b9dfb] into tmpdir=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp, totalSize=62.6 K 2023-03-21 11:55:56,311 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61680 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-21 11:55:56,311 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1912): 1baade11f622687e2c1237b8441afd49/B is initiating minor compaction (all files) 2023-03-21 11:55:56,311 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1baade11f622687e2c1237b8441afd49/B in TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:56,311 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 8214a0171e384bf7875c252e1c55cb53, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=953, earliestPutTs=1679399752889 2023-03-21 11:55:56,311 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/5edf36bacd6d4c51908da851b0d463a2, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/78c678df2e484924a1c5baca85548bd8, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/895440721ca54307a6567a8ece1dd90d, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/f58907ca78ee499ca9f4804151859caa, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/3a64dd6349ab466bb4fb6210b7aab131] into tmpdir=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp, totalSize=60.2 K 2023-03-21 11:55:56,312 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 2de7da9ae5314538a7fe459110acc268, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=967, earliestPutTs=1679399753014 2023-03-21 11:55:56,312 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 5edf36bacd6d4c51908da851b0d463a2, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=953, earliestPutTs=1679399752889 2023-03-21 11:55:56,312 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 9aa37aa540cd47f4bdbcc8607923a1a3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=987, earliestPutTs=1679399753642 2023-03-21 11:55:56,313 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 78c678df2e484924a1c5baca85548bd8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=967, earliestPutTs=1679399753014 2023-03-21 11:55:56,313 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting b4f70c225a65499c8e4cc23f04c9e3ac, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1002, earliestPutTs=1679399754284 2023-03-21 11:55:56,313 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 895440721ca54307a6567a8ece1dd90d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=987, earliestPutTs=1679399753642 2023-03-21 11:55:56,313 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting ac60d45f5967445f9ffec2d6410b9dfb, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=1014, earliestPutTs=1679399755400 2023-03-21 11:55:56,314 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting f58907ca78ee499ca9f4804151859caa, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1002, earliestPutTs=1679399754284 2023-03-21 11:55:56,314 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 3a64dd6349ab466bb4fb6210b7aab131, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1014, earliestPutTs=1679399755400 2023-03-21 11:55:56,341 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] throttle.PressureAwareThroughputController(145): 1baade11f622687e2c1237b8441afd49#A#compaction#81 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-21 11:55:56,359 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] throttle.PressureAwareThroughputController(145): 1baade11f622687e2c1237b8441afd49#B#compaction#82 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-21 11:55:56,390 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:55:56,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:55:56,392 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:55:56,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:55:56,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:55:56,393 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:55:56,393 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:56,393 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:55:56,393 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:55:56,393 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:56,393 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:56,394 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:56,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:56,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:56,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:56,394 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:56,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:55:56,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:56,395 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:55:56,395 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:56,395 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:55:56,395 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:55:56,395 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:55:56,395 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:55:56,395 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:55:56,395 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:55:56,395 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:55:56,395 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:56,395 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. started... 2023-03-21 11:55:56,396 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1baade11f622687e2c1237b8441afd49 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-21 11:55:56,396 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1baade11f622687e2c1237b8441afd49, store=A 2023-03-21 11:55:56,396 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:56,396 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:55:56,396 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:55:56,396 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1baade11f622687e2c1237b8441afd49, store=B 2023-03-21 11:55:56,396 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:55:56,396 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1baade11f622687e2c1237b8441afd49, store=C 2023-03-21 11:55:56,396 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:55:56,410 DEBUG [Thread-152] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x31222f6c to 127.0.0.1:61761 2023-03-21 11:55:56,410 DEBUG [Thread-152] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-21 11:55:56,413 DEBUG [Thread-154] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x29fd8399 to 127.0.0.1:61761 2023-03-21 11:55:56,413 DEBUG [Thread-154] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-21 11:55:56,417 DEBUG [Thread-158] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7746d7a7 to 127.0.0.1:61761 2023-03-21 11:55:56,418 DEBUG [Thread-158] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-21 11:55:56,422 DEBUG [Thread-156] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x57966649 to 127.0.0.1:61761 2023-03-21 11:55:56,422 DEBUG [Thread-156] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-21 11:55:56,423 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=1034 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/A/6abd6039ab824a9a9205c07e84a3c68a 2023-03-21 11:55:56,434 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=1034 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/B/2ed9f85fc0af424d85f0ae983a91704f 2023-03-21 11:55:56,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(9158): Flush requested on 1baade11f622687e2c1237b8441afd49 2023-03-21 11:55:56,552 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. as already flushing 2023-03-21 11:55:56,552 DEBUG [Thread-141] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x78c6cbc8 to 127.0.0.1:61761 2023-03-21 11:55:56,552 DEBUG [Thread-145] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3339866b to 127.0.0.1:61761 2023-03-21 11:55:56,552 DEBUG [Thread-147] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1ee17908 to 127.0.0.1:61761 2023-03-21 11:55:56,552 DEBUG [Thread-145] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-21 11:55:56,552 DEBUG [Thread-141] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-21 11:55:56,552 DEBUG [Thread-147] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-21 11:55:56,770 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/A/ff706fc77259491c942ed487d93ab2e3 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/ff706fc77259491c942ed487d93ab2e3 2023-03-21 11:55:56,774 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/B/c6c0decdf7774c2eb1b3945823e74440 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/c6c0decdf7774c2eb1b3945823e74440 2023-03-21 11:55:56,776 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 1baade11f622687e2c1237b8441afd49/A of 1baade11f622687e2c1237b8441afd49 into ff706fc77259491c942ed487d93ab2e3(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-21 11:55:56,776 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1baade11f622687e2c1237b8441afd49: 2023-03-21 11:55:56,776 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49., storeName=1baade11f622687e2c1237b8441afd49/A, priority=11, startTime=1679399756308; duration=0sec 2023-03-21 11:55:56,776 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-21 11:55:56,776 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-21 11:55:56,778 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61680 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-21 11:55:56,778 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1912): 1baade11f622687e2c1237b8441afd49/C is initiating minor compaction (all files) 2023-03-21 11:55:56,778 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 1baade11f622687e2c1237b8441afd49/C in TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:56,778 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/a2dc6bb92bac481c8458e6d8d016f6a1, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/b9382c6a30d740e1817cf96c59500eea, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/0aed0c6743ac423abcc70c821d9f2035, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/eeb55826ed4c4c6dbb5f6bd990fc7bd8, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/054815745f9844c191e732d28da3d26e] into tmpdir=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp, totalSize=60.2 K 2023-03-21 11:55:56,778 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting a2dc6bb92bac481c8458e6d8d016f6a1, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=953, earliestPutTs=1679399752889 2023-03-21 11:55:56,779 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting b9382c6a30d740e1817cf96c59500eea, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=967, earliestPutTs=1679399753014 2023-03-21 11:55:56,779 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 0aed0c6743ac423abcc70c821d9f2035, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=987, earliestPutTs=1679399753642 2023-03-21 11:55:56,779 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting eeb55826ed4c4c6dbb5f6bd990fc7bd8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1002, earliestPutTs=1679399754284 2023-03-21 11:55:56,779 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 1baade11f622687e2c1237b8441afd49/B of 1baade11f622687e2c1237b8441afd49 into c6c0decdf7774c2eb1b3945823e74440(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-21 11:55:56,779 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1baade11f622687e2c1237b8441afd49: 2023-03-21 11:55:56,779 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49., storeName=1baade11f622687e2c1237b8441afd49/B, priority=11, startTime=1679399756308; duration=0sec 2023-03-21 11:55:56,780 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 054815745f9844c191e732d28da3d26e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1014, earliestPutTs=1679399755400 2023-03-21 11:55:56,780 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-21 11:55:56,788 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] throttle.PressureAwareThroughputController(145): 1baade11f622687e2c1237b8441afd49#C#compaction#86 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-21 11:55:56,848 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=1034 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/C/39503c92aba64982914b2176c6536307 2023-03-21 11:55:56,854 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/A/6abd6039ab824a9a9205c07e84a3c68a as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/6abd6039ab824a9a9205c07e84a3c68a 2023-03-21 11:55:56,861 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/6abd6039ab824a9a9205c07e84a3c68a, entries=150, sequenceid=1034, filesize=11.9 K 2023-03-21 11:55:56,861 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/B/2ed9f85fc0af424d85f0ae983a91704f as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/2ed9f85fc0af424d85f0ae983a91704f 2023-03-21 11:55:56,868 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/2ed9f85fc0af424d85f0ae983a91704f, entries=150, sequenceid=1034, filesize=11.9 K 2023-03-21 11:55:56,869 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/C/39503c92aba64982914b2176c6536307 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/39503c92aba64982914b2176c6536307 2023-03-21 11:55:56,877 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/39503c92aba64982914b2176c6536307, entries=150, sequenceid=1034, filesize=11.9 K 2023-03-21 11:55:56,878 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=20.13 KB/20610 for 1baade11f622687e2c1237b8441afd49 in 482ms, sequenceid=1034, compaction requested=false 2023-03-21 11:55:56,878 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1baade11f622687e2c1237b8441afd49: 2023-03-21 11:55:56,878 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:55:56,878 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:55:56,878 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:55:56,878 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:56,878 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:55:56,878 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:55:56,880 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:56,880 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:56,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:56,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:56,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:56,880 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:56,880 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:55:56,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:56,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:56,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:56,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:56,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:56,881 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:55:56,882 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:55:56,882 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@747cabec[Count = 0] remaining members to acquire global barrier 2023-03-21 11:55:56,882 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:56,882 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:56,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:56,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:56,883 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:55:56,883 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:55:56,883 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:55:56,883 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:56,883 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:55:56,883 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:56,883 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:55:56,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:56,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:56,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:56,883 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:56,884 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:55:56,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:56,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:56,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:56,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:56,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:56,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:56,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:56,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:55:56,886 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:55:56,886 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:55:56,886 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:55:56,886 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:55:56,886 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:55:56,886 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:55:56,887 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:56,887 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:56,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:56,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:56,887 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:56,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:56,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:55:56,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:55:56,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:56,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:56,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:55:56,888 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:56,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:56,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:56,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:55:56,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:56,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:56,889 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:56,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:55:56,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:55:56,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:56,891 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:56,891 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:55:56,891 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:56,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:55:56,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:55:56,891 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:55:56,891 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:55:56,891 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:55:56,891 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:55:56,891 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:55:56,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:55:56,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:55:56,891 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:55:56,891 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:56,891 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:55:56,891 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:55:56,891 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:55:56,891 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:55:56,992 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:55:56,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:55:57,203 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/C/934160462cd54bca904281cabdff03d5 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/934160462cd54bca904281cabdff03d5 2023-03-21 11:55:57,209 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 1baade11f622687e2c1237b8441afd49/C of 1baade11f622687e2c1237b8441afd49 into 934160462cd54bca904281cabdff03d5(size=12.8 K), total size for store is 24.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-21 11:55:57,209 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1baade11f622687e2c1237b8441afd49: 2023-03-21 11:55:57,209 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49., storeName=1baade11f622687e2c1237b8441afd49/C, priority=11, startTime=1679399756308; duration=0sec 2023-03-21 11:55:57,209 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-21 11:56:02,014 DEBUG [Thread-143] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3b15648c to 127.0.0.1:61761 2023-03-21 11:56:02,014 DEBUG [Thread-143] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-21 11:56:02,030 DEBUG [Thread-149] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x40f66fa6 to 127.0.0.1:61761 2023-03-21 11:56:02,031 DEBUG [Thread-149] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-21 11:56:02,031 INFO [Listener at localhost.localdomain/37475] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-03-21 11:56:02,031 INFO [Listener at localhost.localdomain/37475] hbase.AcidGuaranteesTestTool(394): wrote 206 2023-03-21 11:56:02,031 INFO [Listener at localhost.localdomain/37475] hbase.AcidGuaranteesTestTool(394): wrote 181 2023-03-21 11:56:02,031 INFO [Listener at localhost.localdomain/37475] hbase.AcidGuaranteesTestTool(394): wrote 220 2023-03-21 11:56:02,031 INFO [Listener at localhost.localdomain/37475] hbase.AcidGuaranteesTestTool(394): wrote 211 2023-03-21 11:56:02,031 INFO [Listener at localhost.localdomain/37475] hbase.AcidGuaranteesTestTool(394): wrote 136 2023-03-21 11:56:02,031 INFO [Listener at localhost.localdomain/37475] hbase.AcidGuaranteesTestTool(396): Readers: 2023-03-21 11:56:02,031 INFO [Listener at localhost.localdomain/37475] hbase.AcidGuaranteesTestTool(398): read 4720 2023-03-21 11:56:02,031 INFO [Listener at localhost.localdomain/37475] hbase.AcidGuaranteesTestTool(398): read 4585 2023-03-21 11:56:02,031 INFO [Listener at localhost.localdomain/37475] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-03-21 11:56:02,031 INFO [Listener at localhost.localdomain/37475] hbase.AcidGuaranteesTestTool(402): scanned 2024 2023-03-21 11:56:02,031 INFO [Listener at localhost.localdomain/37475] hbase.AcidGuaranteesTestTool(403): verified 6062 rows 2023-03-21 11:56:02,031 INFO [Listener at localhost.localdomain/37475] hbase.AcidGuaranteesTestTool(402): scanned 2035 2023-03-21 11:56:02,032 INFO [Listener at localhost.localdomain/37475] hbase.AcidGuaranteesTestTool(403): verified 6095 rows 2023-03-21 11:56:02,032 INFO [Listener at localhost.localdomain/37475] client.ConnectionImplementation(1959): Closing master protocol: MasterService 2023-03-21 11:56:02,032 DEBUG [Listener at localhost.localdomain/37475] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4a427c73 to 127.0.0.1:61761 2023-03-21 11:56:02,032 DEBUG [Listener at localhost.localdomain/37475] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-21 11:56:02,036 INFO [Listener at localhost.localdomain/37475] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-03-21 11:56:02,041 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.HMaster$11(2418): Client=jenkins//148.251.75.209 disable TestAcidGuarantees 2023-03-21 11:56:02,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure2.ProcedureExecutor(1029): Stored pid=12, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-03-21 11:56:02,051 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679399762051"}]},"ts":"1679399762051"} 2023-03-21 11:56:02,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-03-21 11:56:02,052 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-03-21 11:56:02,053 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-03-21 11:56:02,054 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=1baade11f622687e2c1237b8441afd49, UNASSIGN}] 2023-03-21 11:56:02,055 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=1baade11f622687e2c1237b8441afd49, UNASSIGN 2023-03-21 11:56:02,056 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=1baade11f622687e2c1237b8441afd49, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:02,056 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679399762056"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1679399762056"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679399762056"}]},"ts":"1679399762056"} 2023-03-21 11:56:02,057 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=14, ppid=13, state=RUNNABLE; CloseRegionProcedure 1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030}] 2023-03-21 11:56:02,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-03-21 11:56:02,214 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 1baade11f622687e2c1237b8441afd49 2023-03-21 11:56:02,214 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 1baade11f622687e2c1237b8441afd49, disabling compactions & flushes 2023-03-21 11:56:02,214 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:56:02,214 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:56:02,215 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. after waiting 0 ms 2023-03-21 11:56:02,215 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:56:02,215 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing 1baade11f622687e2c1237b8441afd49 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-03-21 11:56:02,215 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1baade11f622687e2c1237b8441afd49, store=A 2023-03-21 11:56:02,215 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:56:02,215 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1baade11f622687e2c1237b8441afd49, store=B 2023-03-21 11:56:02,215 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:56:02,215 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1baade11f622687e2c1237b8441afd49, store=C 2023-03-21 11:56:02,215 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:56:02,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-03-21 11:56:02,623 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=1045 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/A/64d1e7638ec84c3f9165203e4fd415b5 2023-03-21 11:56:02,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-03-21 11:56:03,041 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=1045 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/B/e8936b828e1748ad9fce8c664a698338 2023-03-21 11:56:03,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-03-21 11:56:03,460 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=1045 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/C/64a6f98843e2448c8df0797241e93420 2023-03-21 11:56:03,468 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/A/64d1e7638ec84c3f9165203e4fd415b5 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/64d1e7638ec84c3f9165203e4fd415b5 2023-03-21 11:56:03,474 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/64d1e7638ec84c3f9165203e4fd415b5, entries=100, sequenceid=1045, filesize=9.5 K 2023-03-21 11:56:03,475 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/B/e8936b828e1748ad9fce8c664a698338 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/e8936b828e1748ad9fce8c664a698338 2023-03-21 11:56:03,480 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/e8936b828e1748ad9fce8c664a698338, entries=100, sequenceid=1045, filesize=9.5 K 2023-03-21 11:56:03,481 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/.tmp/C/64a6f98843e2448c8df0797241e93420 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/64a6f98843e2448c8df0797241e93420 2023-03-21 11:56:03,489 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/64a6f98843e2448c8df0797241e93420, entries=100, sequenceid=1045, filesize=9.5 K 2023-03-21 11:56:03,490 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=0 B/0 for 1baade11f622687e2c1237b8441afd49 in 1275ms, sequenceid=1045, compaction requested=true 2023-03-21 11:56:03,516 DEBUG [StoreCloser-TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/1ab8e06c6cf6468db0b6a9f3d60d1717, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/779facedcefe4862b469363e0301cc77, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/428d7d3148644c638ab53ce30e985317, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/081f2d6b58cc4ab59e75485f1a78c819, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/116e34b7be594bb88be70982db0eec91, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/84b1b59a8e8d4ca8b78131c9edca2b4b, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/e5f868adc3ec45018d376cfef4d45689, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/dcee3246ab7c4e6293e9f95772fa76bd, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/ae55ceef96844e07b1362851cdc0d7a0, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/6eb0e3139e9743119d8eb5541ba69c5c, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/7520b7ea1bbf4a8d8b901fb9700d9d48, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/3e450e6d72af4f6f8a153fea77e3623c, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/4fbc9568fd394997823c8df520ccf30f, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/b6a8ecdb2ea64b988803ff9a0d55541a, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/d0e42e5a19294e20a53f3d1dba643fe3, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/994a302de1c44da3beccbb1e98a25514, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/33cf0b3361084c078149f0e649518554, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/cbc30cb2011d42d78d763b7fcb2fcba6, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/00f15a59a24247b8b61cd5360a185ee8, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/e1105cf8e4b8435695ab118ee62c63a2, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/fbe1f9e6e4cd4d9fa6c24b9ffa49c223, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/2aa8b2fc68d54ec4846aa1139bb20595, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/8214a0171e384bf7875c252e1c55cb53, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/2de7da9ae5314538a7fe459110acc268, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/9aa37aa540cd47f4bdbcc8607923a1a3, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/b4f70c225a65499c8e4cc23f04c9e3ac, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/ac60d45f5967445f9ffec2d6410b9dfb] to archive 2023-03-21 11:56:03,519 DEBUG [StoreCloser-TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-21 11:56:03,525 DEBUG [StoreCloser-TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/1ab8e06c6cf6468db0b6a9f3d60d1717 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/1ab8e06c6cf6468db0b6a9f3d60d1717 2023-03-21 11:56:03,526 DEBUG [StoreCloser-TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/779facedcefe4862b469363e0301cc77 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/779facedcefe4862b469363e0301cc77 2023-03-21 11:56:03,528 DEBUG [StoreCloser-TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/428d7d3148644c638ab53ce30e985317 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/428d7d3148644c638ab53ce30e985317 2023-03-21 11:56:03,529 DEBUG [StoreCloser-TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/081f2d6b58cc4ab59e75485f1a78c819 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/081f2d6b58cc4ab59e75485f1a78c819 2023-03-21 11:56:03,530 DEBUG [StoreCloser-TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/116e34b7be594bb88be70982db0eec91 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/116e34b7be594bb88be70982db0eec91 2023-03-21 11:56:03,534 DEBUG [StoreCloser-TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/84b1b59a8e8d4ca8b78131c9edca2b4b to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/84b1b59a8e8d4ca8b78131c9edca2b4b 2023-03-21 11:56:03,537 DEBUG [StoreCloser-TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/e5f868adc3ec45018d376cfef4d45689 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/e5f868adc3ec45018d376cfef4d45689 2023-03-21 11:56:03,538 DEBUG [StoreCloser-TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/dcee3246ab7c4e6293e9f95772fa76bd to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/dcee3246ab7c4e6293e9f95772fa76bd 2023-03-21 11:56:03,540 DEBUG [StoreCloser-TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/ae55ceef96844e07b1362851cdc0d7a0 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/ae55ceef96844e07b1362851cdc0d7a0 2023-03-21 11:56:03,541 DEBUG [StoreCloser-TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/6eb0e3139e9743119d8eb5541ba69c5c to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/6eb0e3139e9743119d8eb5541ba69c5c 2023-03-21 11:56:03,543 DEBUG [StoreCloser-TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/7520b7ea1bbf4a8d8b901fb9700d9d48 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/7520b7ea1bbf4a8d8b901fb9700d9d48 2023-03-21 11:56:03,544 DEBUG [StoreCloser-TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/3e450e6d72af4f6f8a153fea77e3623c to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/3e450e6d72af4f6f8a153fea77e3623c 2023-03-21 11:56:03,545 DEBUG [StoreCloser-TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/4fbc9568fd394997823c8df520ccf30f to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/4fbc9568fd394997823c8df520ccf30f 2023-03-21 11:56:03,546 DEBUG [StoreCloser-TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/b6a8ecdb2ea64b988803ff9a0d55541a to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/b6a8ecdb2ea64b988803ff9a0d55541a 2023-03-21 11:56:03,548 DEBUG [StoreCloser-TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/d0e42e5a19294e20a53f3d1dba643fe3 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/d0e42e5a19294e20a53f3d1dba643fe3 2023-03-21 11:56:03,549 DEBUG [StoreCloser-TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/994a302de1c44da3beccbb1e98a25514 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/994a302de1c44da3beccbb1e98a25514 2023-03-21 11:56:03,550 DEBUG [StoreCloser-TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/33cf0b3361084c078149f0e649518554 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/33cf0b3361084c078149f0e649518554 2023-03-21 11:56:03,551 DEBUG [StoreCloser-TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/cbc30cb2011d42d78d763b7fcb2fcba6 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/cbc30cb2011d42d78d763b7fcb2fcba6 2023-03-21 11:56:03,553 DEBUG [StoreCloser-TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/00f15a59a24247b8b61cd5360a185ee8 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/00f15a59a24247b8b61cd5360a185ee8 2023-03-21 11:56:03,555 DEBUG [StoreCloser-TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/e1105cf8e4b8435695ab118ee62c63a2 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/e1105cf8e4b8435695ab118ee62c63a2 2023-03-21 11:56:03,556 DEBUG [StoreCloser-TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/fbe1f9e6e4cd4d9fa6c24b9ffa49c223 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/fbe1f9e6e4cd4d9fa6c24b9ffa49c223 2023-03-21 11:56:03,558 DEBUG [StoreCloser-TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/2aa8b2fc68d54ec4846aa1139bb20595 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/2aa8b2fc68d54ec4846aa1139bb20595 2023-03-21 11:56:03,560 DEBUG [StoreCloser-TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/8214a0171e384bf7875c252e1c55cb53 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/8214a0171e384bf7875c252e1c55cb53 2023-03-21 11:56:03,561 DEBUG [StoreCloser-TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/2de7da9ae5314538a7fe459110acc268 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/2de7da9ae5314538a7fe459110acc268 2023-03-21 11:56:03,566 DEBUG [StoreCloser-TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/9aa37aa540cd47f4bdbcc8607923a1a3 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/9aa37aa540cd47f4bdbcc8607923a1a3 2023-03-21 11:56:03,568 DEBUG [StoreCloser-TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/b4f70c225a65499c8e4cc23f04c9e3ac to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/b4f70c225a65499c8e4cc23f04c9e3ac 2023-03-21 11:56:03,569 DEBUG [StoreCloser-TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/ac60d45f5967445f9ffec2d6410b9dfb to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/ac60d45f5967445f9ffec2d6410b9dfb 2023-03-21 11:56:03,601 DEBUG [StoreCloser-TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/0120925148ae4c879ca049603ff834e6, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/da21ab7ad4974bdf929c76397cb2653b, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/7ce5a963f01b477e988bd7787b392c48, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/ac41af5f10f0440bba4a366f5df13aeb, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/dca0897dc2be49dc98dd161927497655, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/8446248109cd4bc484adc1e36995b117, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/a9e65d306c2d439d8adece1adacca899, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/bb9bdec46c2348a289d415ae05de9ae2, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/e6fbe49c71ea4a30b61e71867fb7a879, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/d93b8a06415240c99c27a1613c713c3f, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/abcc8dbaa4504506ac0769fec5df6e44, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/83de8f130b27469f95deb08d73b65e0e, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/a32bbbc70d54418188b27b433a4885a2, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/ae65b0e384ad4358a63dc9ab53d5e9f7, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/c4e5686cd5d4489b9a934e20f9b3827b, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/9653f09670654169b0bb33dc6a2431a5, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/ddb7e67da9ac4fa594352481b78512c1, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/b7286c70fb974fee938b24e92badf461, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/9b1dfc4ceaf7414aba148680881a7954, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/c2384b6bab7d4c9f970557bc990d65ce, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/9d5003fc12d1450ba50020cd62f33df1, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/5edf36bacd6d4c51908da851b0d463a2, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/37d83a86b3f7491eb22a9364f1ad4b35, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/78c678df2e484924a1c5baca85548bd8, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/895440721ca54307a6567a8ece1dd90d, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/f58907ca78ee499ca9f4804151859caa, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/3a64dd6349ab466bb4fb6210b7aab131] to archive 2023-03-21 11:56:03,602 DEBUG [StoreCloser-TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-21 11:56:03,604 DEBUG [StoreCloser-TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/0120925148ae4c879ca049603ff834e6 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/0120925148ae4c879ca049603ff834e6 2023-03-21 11:56:03,606 DEBUG [StoreCloser-TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/da21ab7ad4974bdf929c76397cb2653b to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/da21ab7ad4974bdf929c76397cb2653b 2023-03-21 11:56:03,610 DEBUG [StoreCloser-TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/7ce5a963f01b477e988bd7787b392c48 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/7ce5a963f01b477e988bd7787b392c48 2023-03-21 11:56:03,611 DEBUG [StoreCloser-TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/ac41af5f10f0440bba4a366f5df13aeb to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/ac41af5f10f0440bba4a366f5df13aeb 2023-03-21 11:56:03,613 DEBUG [StoreCloser-TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/dca0897dc2be49dc98dd161927497655 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/dca0897dc2be49dc98dd161927497655 2023-03-21 11:56:03,614 DEBUG [StoreCloser-TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/8446248109cd4bc484adc1e36995b117 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/8446248109cd4bc484adc1e36995b117 2023-03-21 11:56:03,616 DEBUG [StoreCloser-TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/a9e65d306c2d439d8adece1adacca899 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/a9e65d306c2d439d8adece1adacca899 2023-03-21 11:56:03,617 DEBUG [StoreCloser-TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/bb9bdec46c2348a289d415ae05de9ae2 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/bb9bdec46c2348a289d415ae05de9ae2 2023-03-21 11:56:03,619 DEBUG [StoreCloser-TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/e6fbe49c71ea4a30b61e71867fb7a879 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/e6fbe49c71ea4a30b61e71867fb7a879 2023-03-21 11:56:03,620 DEBUG [StoreCloser-TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/d93b8a06415240c99c27a1613c713c3f to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/d93b8a06415240c99c27a1613c713c3f 2023-03-21 11:56:03,621 DEBUG [StoreCloser-TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/abcc8dbaa4504506ac0769fec5df6e44 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/abcc8dbaa4504506ac0769fec5df6e44 2023-03-21 11:56:03,622 DEBUG [StoreCloser-TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/83de8f130b27469f95deb08d73b65e0e to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/83de8f130b27469f95deb08d73b65e0e 2023-03-21 11:56:03,623 DEBUG [StoreCloser-TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/a32bbbc70d54418188b27b433a4885a2 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/a32bbbc70d54418188b27b433a4885a2 2023-03-21 11:56:03,624 DEBUG [StoreCloser-TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/ae65b0e384ad4358a63dc9ab53d5e9f7 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/ae65b0e384ad4358a63dc9ab53d5e9f7 2023-03-21 11:56:03,625 DEBUG [StoreCloser-TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/c4e5686cd5d4489b9a934e20f9b3827b to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/c4e5686cd5d4489b9a934e20f9b3827b 2023-03-21 11:56:03,626 DEBUG [StoreCloser-TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/9653f09670654169b0bb33dc6a2431a5 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/9653f09670654169b0bb33dc6a2431a5 2023-03-21 11:56:03,628 DEBUG [StoreCloser-TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/ddb7e67da9ac4fa594352481b78512c1 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/ddb7e67da9ac4fa594352481b78512c1 2023-03-21 11:56:03,629 DEBUG [StoreCloser-TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/b7286c70fb974fee938b24e92badf461 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/b7286c70fb974fee938b24e92badf461 2023-03-21 11:56:03,630 DEBUG [StoreCloser-TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/9b1dfc4ceaf7414aba148680881a7954 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/9b1dfc4ceaf7414aba148680881a7954 2023-03-21 11:56:03,631 DEBUG [StoreCloser-TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/c2384b6bab7d4c9f970557bc990d65ce to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/c2384b6bab7d4c9f970557bc990d65ce 2023-03-21 11:56:03,633 DEBUG [StoreCloser-TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/9d5003fc12d1450ba50020cd62f33df1 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/9d5003fc12d1450ba50020cd62f33df1 2023-03-21 11:56:03,634 DEBUG [StoreCloser-TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/5edf36bacd6d4c51908da851b0d463a2 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/5edf36bacd6d4c51908da851b0d463a2 2023-03-21 11:56:03,635 DEBUG [StoreCloser-TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/37d83a86b3f7491eb22a9364f1ad4b35 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/37d83a86b3f7491eb22a9364f1ad4b35 2023-03-21 11:56:03,636 DEBUG [StoreCloser-TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/78c678df2e484924a1c5baca85548bd8 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/78c678df2e484924a1c5baca85548bd8 2023-03-21 11:56:03,637 DEBUG [StoreCloser-TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/895440721ca54307a6567a8ece1dd90d to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/895440721ca54307a6567a8ece1dd90d 2023-03-21 11:56:03,638 DEBUG [StoreCloser-TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/f58907ca78ee499ca9f4804151859caa to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/f58907ca78ee499ca9f4804151859caa 2023-03-21 11:56:03,639 DEBUG [StoreCloser-TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/3a64dd6349ab466bb4fb6210b7aab131 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/3a64dd6349ab466bb4fb6210b7aab131 2023-03-21 11:56:03,642 DEBUG [StoreCloser-TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/d238a1367a424afcaa9c605131083402, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/dd688aa38f3845188d8c91c8f59da958, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/37682e8f339d4314b6b8a02999b5c9fd, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/8c5c3e47186c4063b36028ad05769e5d, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/9dc2aed25ace42218915e42bd296a96f, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/d6252139523742b1ae6cc1cc4414ceca, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/82afd4b6f2a846a9af5f125f12c5fc45, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/24d0729fddcc4536a16ba5ed97d23127, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/dbca2fc359834f2dbe37ff32cf1578eb, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/b76e30e438d0471e999a4b13dde5d7f6, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/5ff8d939f3494163bf8bbe3f10e94e84, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/45d08817049145c087900115258424d4, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/480e1800afbc43c79262c58c534461fd, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/6db913c5a4c2484a995871fb7be47e9d, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/5e51e46a266d459980b796be1813f2ff, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/7f32ec67ab3f4ec8afadfc498bc63068, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/cdf2cd8db0234c51ad34438134e69923, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/f9de91cdac0944069e36c00cb76e0a3b, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/5193289533354700ada1dddf509941dd, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/77e35199ff9b4a9a8c87ed3753716393, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/56a5d0aae2b04c5fb9fae7138cc79169, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/a2dc6bb92bac481c8458e6d8d016f6a1, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/fbdbac0fa4d24763b5e55625ff3a5b4a, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/b9382c6a30d740e1817cf96c59500eea, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/0aed0c6743ac423abcc70c821d9f2035, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/eeb55826ed4c4c6dbb5f6bd990fc7bd8, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/054815745f9844c191e732d28da3d26e] to archive 2023-03-21 11:56:03,643 DEBUG [StoreCloser-TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-21 11:56:03,645 DEBUG [StoreCloser-TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/d238a1367a424afcaa9c605131083402 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/d238a1367a424afcaa9c605131083402 2023-03-21 11:56:03,646 DEBUG [StoreCloser-TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/dd688aa38f3845188d8c91c8f59da958 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/dd688aa38f3845188d8c91c8f59da958 2023-03-21 11:56:03,647 DEBUG [StoreCloser-TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/37682e8f339d4314b6b8a02999b5c9fd to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/37682e8f339d4314b6b8a02999b5c9fd 2023-03-21 11:56:03,648 DEBUG [StoreCloser-TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/8c5c3e47186c4063b36028ad05769e5d to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/8c5c3e47186c4063b36028ad05769e5d 2023-03-21 11:56:03,649 DEBUG [StoreCloser-TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/9dc2aed25ace42218915e42bd296a96f to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/9dc2aed25ace42218915e42bd296a96f 2023-03-21 11:56:03,650 DEBUG [StoreCloser-TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/d6252139523742b1ae6cc1cc4414ceca to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/d6252139523742b1ae6cc1cc4414ceca 2023-03-21 11:56:03,651 DEBUG [StoreCloser-TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/82afd4b6f2a846a9af5f125f12c5fc45 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/82afd4b6f2a846a9af5f125f12c5fc45 2023-03-21 11:56:03,651 DEBUG [StoreCloser-TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/24d0729fddcc4536a16ba5ed97d23127 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/24d0729fddcc4536a16ba5ed97d23127 2023-03-21 11:56:03,652 DEBUG [StoreCloser-TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/dbca2fc359834f2dbe37ff32cf1578eb to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/dbca2fc359834f2dbe37ff32cf1578eb 2023-03-21 11:56:03,653 DEBUG [StoreCloser-TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/b76e30e438d0471e999a4b13dde5d7f6 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/b76e30e438d0471e999a4b13dde5d7f6 2023-03-21 11:56:03,654 DEBUG [StoreCloser-TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/5ff8d939f3494163bf8bbe3f10e94e84 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/5ff8d939f3494163bf8bbe3f10e94e84 2023-03-21 11:56:03,655 DEBUG [StoreCloser-TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/45d08817049145c087900115258424d4 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/45d08817049145c087900115258424d4 2023-03-21 11:56:03,657 DEBUG [StoreCloser-TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/480e1800afbc43c79262c58c534461fd to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/480e1800afbc43c79262c58c534461fd 2023-03-21 11:56:03,658 DEBUG [StoreCloser-TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/6db913c5a4c2484a995871fb7be47e9d to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/6db913c5a4c2484a995871fb7be47e9d 2023-03-21 11:56:03,659 DEBUG [StoreCloser-TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/5e51e46a266d459980b796be1813f2ff to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/5e51e46a266d459980b796be1813f2ff 2023-03-21 11:56:03,661 DEBUG [StoreCloser-TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/7f32ec67ab3f4ec8afadfc498bc63068 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/7f32ec67ab3f4ec8afadfc498bc63068 2023-03-21 11:56:03,662 DEBUG [StoreCloser-TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/cdf2cd8db0234c51ad34438134e69923 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/cdf2cd8db0234c51ad34438134e69923 2023-03-21 11:56:03,663 DEBUG [StoreCloser-TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/f9de91cdac0944069e36c00cb76e0a3b to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/f9de91cdac0944069e36c00cb76e0a3b 2023-03-21 11:56:03,665 DEBUG [StoreCloser-TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/5193289533354700ada1dddf509941dd to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/5193289533354700ada1dddf509941dd 2023-03-21 11:56:03,666 DEBUG [StoreCloser-TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/77e35199ff9b4a9a8c87ed3753716393 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/77e35199ff9b4a9a8c87ed3753716393 2023-03-21 11:56:03,667 DEBUG [StoreCloser-TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/56a5d0aae2b04c5fb9fae7138cc79169 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/56a5d0aae2b04c5fb9fae7138cc79169 2023-03-21 11:56:03,668 DEBUG [StoreCloser-TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/a2dc6bb92bac481c8458e6d8d016f6a1 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/a2dc6bb92bac481c8458e6d8d016f6a1 2023-03-21 11:56:03,669 DEBUG [StoreCloser-TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/fbdbac0fa4d24763b5e55625ff3a5b4a to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/fbdbac0fa4d24763b5e55625ff3a5b4a 2023-03-21 11:56:03,671 DEBUG [StoreCloser-TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/b9382c6a30d740e1817cf96c59500eea to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/b9382c6a30d740e1817cf96c59500eea 2023-03-21 11:56:03,672 DEBUG [StoreCloser-TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/0aed0c6743ac423abcc70c821d9f2035 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/0aed0c6743ac423abcc70c821d9f2035 2023-03-21 11:56:03,674 DEBUG [StoreCloser-TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/eeb55826ed4c4c6dbb5f6bd990fc7bd8 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/eeb55826ed4c4c6dbb5f6bd990fc7bd8 2023-03-21 11:56:03,675 DEBUG [StoreCloser-TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/054815745f9844c191e732d28da3d26e to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/054815745f9844c191e732d28da3d26e 2023-03-21 11:56:03,679 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/recovered.edits/1048.seqid, newMaxSeqId=1048, maxSeqId=1 2023-03-21 11:56:03,680 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49. 2023-03-21 11:56:03,680 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 1baade11f622687e2c1237b8441afd49: 2023-03-21 11:56:03,682 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 1baade11f622687e2c1237b8441afd49 2023-03-21 11:56:03,682 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=1baade11f622687e2c1237b8441afd49, regionState=CLOSED 2023-03-21 11:56:03,682 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679399763682"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679399763682"}]},"ts":"1679399763682"} 2023-03-21 11:56:03,684 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=14, resume processing ppid=13 2023-03-21 11:56:03,685 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=13, state=SUCCESS; CloseRegionProcedure 1baade11f622687e2c1237b8441afd49, server=jenkins-hbase20.apache.org,34369,1679399732030 in 1.6260 sec 2023-03-21 11:56:03,686 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=13, resume processing ppid=12 2023-03-21 11:56:03,686 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=1baade11f622687e2c1237b8441afd49, UNASSIGN in 1.6310 sec 2023-03-21 11:56:03,687 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679399763687"}]},"ts":"1679399763687"} 2023-03-21 11:56:03,688 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-03-21 11:56:03,690 INFO [PEWorker-3] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-03-21 11:56:03,692 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=12, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.6480 sec 2023-03-21 11:56:04,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-03-21 11:56:04,158 INFO [Listener at localhost.localdomain/37475] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 12 completed 2023-03-21 11:56:04,162 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.HMaster$5(2228): Client=jenkins//148.251.75.209 delete TestAcidGuarantees 2023-03-21 11:56:04,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure2.ProcedureExecutor(1029): Stored pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-03-21 11:56:04,170 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-21 11:56:04,172 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=15, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-21 11:56:04,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-03-21 11:56:04,176 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/.tmp/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49 2023-03-21 11:56:04,180 DEBUG [HFileArchiver-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/.tmp/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A, FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/.tmp/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B, FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/.tmp/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C, FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/.tmp/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/recovered.edits] 2023-03-21 11:56:04,184 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/.tmp/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/64d1e7638ec84c3f9165203e4fd415b5 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/64d1e7638ec84c3f9165203e4fd415b5 2023-03-21 11:56:04,185 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/.tmp/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/6abd6039ab824a9a9205c07e84a3c68a to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/6abd6039ab824a9a9205c07e84a3c68a 2023-03-21 11:56:04,187 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/.tmp/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/ff706fc77259491c942ed487d93ab2e3 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/A/ff706fc77259491c942ed487d93ab2e3 2023-03-21 11:56:04,190 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/.tmp/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/2ed9f85fc0af424d85f0ae983a91704f to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/2ed9f85fc0af424d85f0ae983a91704f 2023-03-21 11:56:04,191 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/.tmp/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/c6c0decdf7774c2eb1b3945823e74440 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/c6c0decdf7774c2eb1b3945823e74440 2023-03-21 11:56:04,192 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/.tmp/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/e8936b828e1748ad9fce8c664a698338 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/B/e8936b828e1748ad9fce8c664a698338 2023-03-21 11:56:04,195 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/.tmp/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/39503c92aba64982914b2176c6536307 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/39503c92aba64982914b2176c6536307 2023-03-21 11:56:04,196 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/.tmp/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/64a6f98843e2448c8df0797241e93420 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/64a6f98843e2448c8df0797241e93420 2023-03-21 11:56:04,199 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/.tmp/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/934160462cd54bca904281cabdff03d5 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/C/934160462cd54bca904281cabdff03d5 2023-03-21 11:56:04,204 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/.tmp/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/recovered.edits/1048.seqid to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49/recovered.edits/1048.seqid 2023-03-21 11:56:04,205 DEBUG [HFileArchiver-3] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/.tmp/data/default/TestAcidGuarantees/1baade11f622687e2c1237b8441afd49 2023-03-21 11:56:04,205 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-21 11:56:04,208 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=15, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-21 11:56:04,230 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-03-21 11:56:04,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-03-21 11:56:04,276 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-03-21 11:56:04,277 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=15, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-21 11:56:04,277 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-03-21 11:56:04,277 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1679399764277"}]},"ts":"9223372036854775807"} 2023-03-21 11:56:04,280 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-03-21 11:56:04,280 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 1baade11f622687e2c1237b8441afd49, NAME => 'TestAcidGuarantees,,1679399735652.1baade11f622687e2c1237b8441afd49.', STARTKEY => '', ENDKEY => ''}] 2023-03-21 11:56:04,280 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-03-21 11:56:04,281 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1679399764280"}]},"ts":"9223372036854775807"} 2023-03-21 11:56:04,283 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-03-21 11:56:04,285 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=15, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-21 11:56:04,287 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=15, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 123 msec 2023-03-21 11:56:04,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-03-21 11:56:04,475 INFO [Listener at localhost.localdomain/37475] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 15 completed 2023-03-21 11:56:04,494 INFO [Listener at localhost.localdomain/37475] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMixedAtomicity Thread=233 (was 205) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-6 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-2 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x7f4b7c22-shared-pool-21 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x7f4b7c22-shared-pool-22 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-12 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-3 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-1-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-5 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: (jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-9 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-10 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-11 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Parameter Sending Thread #1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-8 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-4 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-13 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RSProcedureDispatcher-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-7 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x7f4b7c22-shared-pool-23 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_202351466_15 at /127.0.0.1:33292 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x7f4b7c22-shared-pool-24 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS:0;jenkins-hbase20:34369-shortCompactions-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:549) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=566 (was 470) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=315 (was 231) - SystemLoadAverage LEAK? -, ProcessCount=167 (was 167), AvailableMemoryMB=9571 (was 9439) - AvailableMemoryMB LEAK? - 2023-03-21 11:56:04,507 INFO [Listener at localhost.localdomain/37475] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobMixedAtomicity Thread=232, OpenFileDescriptor=566, MaxFileDescriptor=60000, SystemLoadAverage=315, ProcessCount=167, AvailableMemoryMB=9570 2023-03-21 11:56:04,509 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-21 11:56:04,509 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.HMaster$4(2112): Client=jenkins//148.251.75.209 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-21 11:56:04,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure2.ProcedureExecutor(1029): Stored pid=16, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-03-21 11:56:04,512 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-03-21 11:56:04,512 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(697): Client=jenkins//148.251.75.209 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 16 2023-03-21 11:56:04,513 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-03-21 11:56:04,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-03-21 11:56:04,514 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/.tmp/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352 2023-03-21 11:56:04,515 DEBUG [HFileArchiver-4] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/.tmp/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352 empty. 2023-03-21 11:56:04,516 DEBUG [HFileArchiver-4] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/.tmp/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352 2023-03-21 11:56:04,516 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-21 11:56:04,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-03-21 11:56:04,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-03-21 11:56:04,929 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-21 11:56:04,930 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 8041406d0a1ddc5b9409edb79457d352, NAME => 'TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/.tmp 2023-03-21 11:56:05,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-03-21 11:56:05,340 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-21 11:56:05,341 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 8041406d0a1ddc5b9409edb79457d352, disabling compactions & flushes 2023-03-21 11:56:05,341 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. 2023-03-21 11:56:05,341 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. 2023-03-21 11:56:05,341 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. after waiting 0 ms 2023-03-21 11:56:05,341 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. 2023-03-21 11:56:05,341 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. 2023-03-21 11:56:05,341 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 8041406d0a1ddc5b9409edb79457d352: 2023-03-21 11:56:05,347 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-03-21 11:56:05,349 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679399765348"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679399765348"}]},"ts":"1679399765348"} 2023-03-21 11:56:05,350 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-03-21 11:56:05,351 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-03-21 11:56:05,352 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679399765351"}]},"ts":"1679399765351"} 2023-03-21 11:56:05,353 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-03-21 11:56:05,355 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=8041406d0a1ddc5b9409edb79457d352, ASSIGN}] 2023-03-21 11:56:05,357 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=8041406d0a1ddc5b9409edb79457d352, ASSIGN 2023-03-21 11:56:05,358 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=8041406d0a1ddc5b9409edb79457d352, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,34369,1679399732030; forceNewPlan=false, retain=false 2023-03-21 11:56:05,509 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=8041406d0a1ddc5b9409edb79457d352, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:05,509 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679399765509"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1679399765509"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679399765509"}]},"ts":"1679399765509"} 2023-03-21 11:56:05,511 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=18, ppid=17, state=RUNNABLE; OpenRegionProcedure 8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030}] 2023-03-21 11:56:05,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-03-21 11:56:05,667 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. 2023-03-21 11:56:05,667 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 8041406d0a1ddc5b9409edb79457d352, NAME => 'TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352.', STARTKEY => '', ENDKEY => ''} 2023-03-21 11:56:05,667 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 8041406d0a1ddc5b9409edb79457d352 2023-03-21 11:56:05,667 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-21 11:56:05,667 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 8041406d0a1ddc5b9409edb79457d352 2023-03-21 11:56:05,667 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 8041406d0a1ddc5b9409edb79457d352 2023-03-21 11:56:05,669 INFO [StoreOpener-8041406d0a1ddc5b9409edb79457d352-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 8041406d0a1ddc5b9409edb79457d352 2023-03-21 11:56:05,670 DEBUG [StoreOpener-8041406d0a1ddc5b9409edb79457d352-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A 2023-03-21 11:56:05,670 DEBUG [StoreOpener-8041406d0a1ddc5b9409edb79457d352-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A 2023-03-21 11:56:05,670 INFO [StoreOpener-8041406d0a1ddc5b9409edb79457d352-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-21 11:56:05,670 INFO [StoreOpener-8041406d0a1ddc5b9409edb79457d352-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 8041406d0a1ddc5b9409edb79457d352 columnFamilyName A 2023-03-21 11:56:05,671 INFO [StoreOpener-8041406d0a1ddc5b9409edb79457d352-1] regionserver.HStore(310): Store=8041406d0a1ddc5b9409edb79457d352/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-21 11:56:05,671 INFO [StoreOpener-8041406d0a1ddc5b9409edb79457d352-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 8041406d0a1ddc5b9409edb79457d352 2023-03-21 11:56:05,673 DEBUG [StoreOpener-8041406d0a1ddc5b9409edb79457d352-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B 2023-03-21 11:56:05,673 DEBUG [StoreOpener-8041406d0a1ddc5b9409edb79457d352-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B 2023-03-21 11:56:05,673 INFO [StoreOpener-8041406d0a1ddc5b9409edb79457d352-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-21 11:56:05,674 INFO [StoreOpener-8041406d0a1ddc5b9409edb79457d352-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 8041406d0a1ddc5b9409edb79457d352 columnFamilyName B 2023-03-21 11:56:05,674 INFO [StoreOpener-8041406d0a1ddc5b9409edb79457d352-1] regionserver.HStore(310): Store=8041406d0a1ddc5b9409edb79457d352/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-21 11:56:05,674 INFO [StoreOpener-8041406d0a1ddc5b9409edb79457d352-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 8041406d0a1ddc5b9409edb79457d352 2023-03-21 11:56:05,676 DEBUG [StoreOpener-8041406d0a1ddc5b9409edb79457d352-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C 2023-03-21 11:56:05,676 DEBUG [StoreOpener-8041406d0a1ddc5b9409edb79457d352-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C 2023-03-21 11:56:05,676 INFO [StoreOpener-8041406d0a1ddc5b9409edb79457d352-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-21 11:56:05,676 INFO [StoreOpener-8041406d0a1ddc5b9409edb79457d352-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 8041406d0a1ddc5b9409edb79457d352 columnFamilyName C 2023-03-21 11:56:05,677 INFO [StoreOpener-8041406d0a1ddc5b9409edb79457d352-1] regionserver.HStore(310): Store=8041406d0a1ddc5b9409edb79457d352/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-21 11:56:05,677 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. 2023-03-21 11:56:05,678 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352 2023-03-21 11:56:05,678 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352 2023-03-21 11:56:05,680 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-21 11:56:05,682 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 8041406d0a1ddc5b9409edb79457d352 2023-03-21 11:56:05,684 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-21 11:56:05,684 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 8041406d0a1ddc5b9409edb79457d352; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62687852, jitterRate=-0.06587821245193481}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-21 11:56:05,684 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 8041406d0a1ddc5b9409edb79457d352: 2023-03-21 11:56:05,685 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352., pid=18, masterSystemTime=1679399765663 2023-03-21 11:56:05,687 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. 2023-03-21 11:56:05,687 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. 2023-03-21 11:56:05,687 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=8041406d0a1ddc5b9409edb79457d352, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:05,688 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679399765687"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1679399765687"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1679399765687"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1679399765687"}]},"ts":"1679399765687"} 2023-03-21 11:56:05,692 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=18, resume processing ppid=17 2023-03-21 11:56:05,692 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=18, ppid=17, state=SUCCESS; OpenRegionProcedure 8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 in 179 msec 2023-03-21 11:56:05,693 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=17, resume processing ppid=16 2023-03-21 11:56:05,693 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=17, ppid=16, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=8041406d0a1ddc5b9409edb79457d352, ASSIGN in 337 msec 2023-03-21 11:56:05,694 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-03-21 11:56:05,694 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679399765694"}]},"ts":"1679399765694"} 2023-03-21 11:56:05,695 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-03-21 11:56:05,699 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-03-21 11:56:05,700 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=16, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1900 sec 2023-03-21 11:56:06,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-03-21 11:56:06,620 INFO [Listener at localhost.localdomain/37475] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 16 completed 2023-03-21 11:56:06,621 DEBUG [Listener at localhost.localdomain/37475] zookeeper.ReadOnlyZKClient(139): Connect 0x7dffa3dd to 127.0.0.1:61761 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-21 11:56:06,625 DEBUG [Listener at localhost.localdomain/37475] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@17a402d2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-21 11:56:06,627 DEBUG [Listener at localhost.localdomain/37475] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-21 11:56:06,647 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:55144, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-21 11:56:06,649 DEBUG [Listener at localhost.localdomain/37475] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-03-21 11:56:06,653 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:41760, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-03-21 11:56:06,661 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-21 11:56:06,661 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.HMaster$12(2454): Client=jenkins//148.251.75.209 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-21 11:56:06,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure2.ProcedureExecutor(1029): Stored pid=19, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-03-21 11:56:06,715 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-03-21 11:56:06,716 DEBUG [PEWorker-4] util.FSTableDescriptors(519): Deleted hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-21 11:56:06,716 INFO [PEWorker-4] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-03-21 11:56:06,722 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=20, ppid=19, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-03-21 11:56:06,742 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=8041406d0a1ddc5b9409edb79457d352, REOPEN/MOVE}] 2023-03-21 11:56:06,745 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=8041406d0a1ddc5b9409edb79457d352, REOPEN/MOVE 2023-03-21 11:56:06,747 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=8041406d0a1ddc5b9409edb79457d352, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:06,747 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679399766747"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1679399766747"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679399766747"}]},"ts":"1679399766747"} 2023-03-21 11:56:06,754 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=22, ppid=21, state=RUNNABLE; CloseRegionProcedure 8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030}] 2023-03-21 11:56:06,908 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 8041406d0a1ddc5b9409edb79457d352 2023-03-21 11:56:06,908 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 8041406d0a1ddc5b9409edb79457d352, disabling compactions & flushes 2023-03-21 11:56:06,909 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. 2023-03-21 11:56:06,909 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. 2023-03-21 11:56:06,909 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. after waiting 0 ms 2023-03-21 11:56:06,909 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. 2023-03-21 11:56:06,913 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-03-21 11:56:06,913 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. 2023-03-21 11:56:06,913 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 8041406d0a1ddc5b9409edb79457d352: 2023-03-21 11:56:06,913 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(3509): Not adding moved region record: 8041406d0a1ddc5b9409edb79457d352 to self. 2023-03-21 11:56:06,915 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 8041406d0a1ddc5b9409edb79457d352 2023-03-21 11:56:06,916 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=8041406d0a1ddc5b9409edb79457d352, regionState=CLOSED 2023-03-21 11:56:06,916 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679399766916"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679399766916"}]},"ts":"1679399766916"} 2023-03-21 11:56:06,918 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=22, resume processing ppid=21 2023-03-21 11:56:06,918 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=22, ppid=21, state=SUCCESS; CloseRegionProcedure 8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 in 163 msec 2023-03-21 11:56:06,919 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=8041406d0a1ddc5b9409edb79457d352, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase20.apache.org,34369,1679399732030; forceNewPlan=false, retain=true 2023-03-21 11:56:07,069 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=8041406d0a1ddc5b9409edb79457d352, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:07,070 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679399767069"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1679399767069"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679399767069"}]},"ts":"1679399767069"} 2023-03-21 11:56:07,071 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=23, ppid=21, state=RUNNABLE; OpenRegionProcedure 8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030}] 2023-03-21 11:56:07,227 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. 2023-03-21 11:56:07,227 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 8041406d0a1ddc5b9409edb79457d352, NAME => 'TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352.', STARTKEY => '', ENDKEY => ''} 2023-03-21 11:56:07,227 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 8041406d0a1ddc5b9409edb79457d352 2023-03-21 11:56:07,227 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-21 11:56:07,227 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 8041406d0a1ddc5b9409edb79457d352 2023-03-21 11:56:07,227 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 8041406d0a1ddc5b9409edb79457d352 2023-03-21 11:56:07,229 INFO [StoreOpener-8041406d0a1ddc5b9409edb79457d352-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 8041406d0a1ddc5b9409edb79457d352 2023-03-21 11:56:07,231 DEBUG [StoreOpener-8041406d0a1ddc5b9409edb79457d352-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A 2023-03-21 11:56:07,231 DEBUG [StoreOpener-8041406d0a1ddc5b9409edb79457d352-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A 2023-03-21 11:56:07,231 INFO [StoreOpener-8041406d0a1ddc5b9409edb79457d352-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-21 11:56:07,233 INFO [StoreOpener-8041406d0a1ddc5b9409edb79457d352-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 8041406d0a1ddc5b9409edb79457d352 columnFamilyName A 2023-03-21 11:56:07,235 INFO [StoreOpener-8041406d0a1ddc5b9409edb79457d352-1] regionserver.HStore(310): Store=8041406d0a1ddc5b9409edb79457d352/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-21 11:56:07,236 INFO [StoreOpener-8041406d0a1ddc5b9409edb79457d352-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 8041406d0a1ddc5b9409edb79457d352 2023-03-21 11:56:07,237 DEBUG [StoreOpener-8041406d0a1ddc5b9409edb79457d352-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B 2023-03-21 11:56:07,237 DEBUG [StoreOpener-8041406d0a1ddc5b9409edb79457d352-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B 2023-03-21 11:56:07,237 INFO [StoreOpener-8041406d0a1ddc5b9409edb79457d352-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-21 11:56:07,237 INFO [StoreOpener-8041406d0a1ddc5b9409edb79457d352-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 8041406d0a1ddc5b9409edb79457d352 columnFamilyName B 2023-03-21 11:56:07,238 INFO [StoreOpener-8041406d0a1ddc5b9409edb79457d352-1] regionserver.HStore(310): Store=8041406d0a1ddc5b9409edb79457d352/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-21 11:56:07,238 INFO [StoreOpener-8041406d0a1ddc5b9409edb79457d352-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 8041406d0a1ddc5b9409edb79457d352 2023-03-21 11:56:07,239 DEBUG [StoreOpener-8041406d0a1ddc5b9409edb79457d352-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C 2023-03-21 11:56:07,239 DEBUG [StoreOpener-8041406d0a1ddc5b9409edb79457d352-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C 2023-03-21 11:56:07,239 INFO [StoreOpener-8041406d0a1ddc5b9409edb79457d352-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-21 11:56:07,240 INFO [StoreOpener-8041406d0a1ddc5b9409edb79457d352-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 8041406d0a1ddc5b9409edb79457d352 columnFamilyName C 2023-03-21 11:56:07,240 INFO [StoreOpener-8041406d0a1ddc5b9409edb79457d352-1] regionserver.HStore(310): Store=8041406d0a1ddc5b9409edb79457d352/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-21 11:56:07,240 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. 2023-03-21 11:56:07,241 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352 2023-03-21 11:56:07,242 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352 2023-03-21 11:56:07,243 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-21 11:56:07,244 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 8041406d0a1ddc5b9409edb79457d352 2023-03-21 11:56:07,244 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 8041406d0a1ddc5b9409edb79457d352; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67789409, jitterRate=0.010140910744667053}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-21 11:56:07,245 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 8041406d0a1ddc5b9409edb79457d352: 2023-03-21 11:56:07,245 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352., pid=23, masterSystemTime=1679399767223 2023-03-21 11:56:07,246 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. 2023-03-21 11:56:07,247 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. 2023-03-21 11:56:07,247 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=8041406d0a1ddc5b9409edb79457d352, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:07,247 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679399767247"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1679399767247"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1679399767247"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1679399767247"}]},"ts":"1679399767247"} 2023-03-21 11:56:07,250 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=23, resume processing ppid=21 2023-03-21 11:56:07,250 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=23, ppid=21, state=SUCCESS; OpenRegionProcedure 8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 in 177 msec 2023-03-21 11:56:07,251 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=21, resume processing ppid=20 2023-03-21 11:56:07,251 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=21, ppid=20, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=8041406d0a1ddc5b9409edb79457d352, REOPEN/MOVE in 508 msec 2023-03-21 11:56:07,254 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=20, resume processing ppid=19 2023-03-21 11:56:07,254 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=20, ppid=19, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 532 msec 2023-03-21 11:56:07,257 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=19, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 590 msec 2023-03-21 11:56:07,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1227): Checking to see if procedure is done pid=19 2023-03-21 11:56:07,267 DEBUG [Listener at localhost.localdomain/37475] zookeeper.ReadOnlyZKClient(139): Connect 0x58b82304 to 127.0.0.1:61761 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-21 11:56:07,271 DEBUG [Listener at localhost.localdomain/37475] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5499dabb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-21 11:56:07,272 DEBUG [Listener at localhost.localdomain/37475] zookeeper.ReadOnlyZKClient(139): Connect 0x6dc16927 to 127.0.0.1:61761 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-21 11:56:07,290 DEBUG [Listener at localhost.localdomain/37475] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2bf3a99a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-21 11:56:07,290 DEBUG [Listener at localhost.localdomain/37475] zookeeper.ReadOnlyZKClient(139): Connect 0x031d0ca0 to 127.0.0.1:61761 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-21 11:56:07,298 DEBUG [Listener at localhost.localdomain/37475] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6573c5b5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-21 11:56:07,299 DEBUG [Listener at localhost.localdomain/37475] zookeeper.ReadOnlyZKClient(139): Connect 0x029d68f7 to 127.0.0.1:61761 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-21 11:56:07,306 DEBUG [Listener at localhost.localdomain/37475] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@79385b7c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-21 11:56:07,307 DEBUG [Listener at localhost.localdomain/37475] zookeeper.ReadOnlyZKClient(139): Connect 0x6322d500 to 127.0.0.1:61761 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-21 11:56:07,321 DEBUG [Listener at localhost.localdomain/37475] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@a11c343, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-21 11:56:07,322 DEBUG [Listener at localhost.localdomain/37475] zookeeper.ReadOnlyZKClient(139): Connect 0x4af56759 to 127.0.0.1:61761 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-21 11:56:07,331 DEBUG [Listener at localhost.localdomain/37475] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4e7aae9f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-21 11:56:07,332 DEBUG [Listener at localhost.localdomain/37475] zookeeper.ReadOnlyZKClient(139): Connect 0x5d07b2f5 to 127.0.0.1:61761 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-21 11:56:07,340 DEBUG [Listener at localhost.localdomain/37475] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3ca9d9a3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-21 11:56:07,340 DEBUG [Listener at localhost.localdomain/37475] zookeeper.ReadOnlyZKClient(139): Connect 0x5446e9d6 to 127.0.0.1:61761 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-21 11:56:07,346 DEBUG [Listener at localhost.localdomain/37475] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@34b669b1, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-21 11:56:07,347 DEBUG [Listener at localhost.localdomain/37475] zookeeper.ReadOnlyZKClient(139): Connect 0x05e8b710 to 127.0.0.1:61761 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-21 11:56:07,356 DEBUG [Listener at localhost.localdomain/37475] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6683e622, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-21 11:56:07,362 DEBUG [hconnection-0x4fd65ea2-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-21 11:56:07,368 DEBUG [hconnection-0x79bf056-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-21 11:56:07,368 DEBUG [hconnection-0x6fd022bf-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-21 11:56:07,368 DEBUG [hconnection-0x75e078d0-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-21 11:56:07,369 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:56:07,370 DEBUG [hconnection-0x5125da27-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-21 11:56:07,372 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:55146, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-21 11:56:07,372 DEBUG [hconnection-0xf037b30-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-21 11:56:07,376 DEBUG [hconnection-0x41f5a3f0-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-21 11:56:07,378 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:55180, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-21 11:56:07,372 DEBUG [hconnection-0x34aa473d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-21 11:56:07,383 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:55148, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-21 11:56:07,376 DEBUG [hconnection-0x6393ae7c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-21 11:56:07,376 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:55166, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-21 11:56:07,386 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:55188, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-21 11:56:07,384 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:55200, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-21 11:56:07,388 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:55190, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-21 11:56:07,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:56:07,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:56:07,389 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:56:07,389 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:07,389 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:56:07,389 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:56:07,390 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:07,390 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:07,390 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:55216, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-21 11:56:07,393 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:07,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:07,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:07,395 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:07,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:07,399 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:07,400 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:56:07,400 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:07,400 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:55208, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-21 11:56:07,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:56:07,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:07,401 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:56:07,402 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:56:07,402 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:56:07,402 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:56:07,402 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:07,402 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:56:07,402 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:56:07,403 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:56:07,403 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. 2023-03-21 11:56:07,403 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. started... 2023-03-21 11:56:07,412 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8041406d0a1ddc5b9409edb79457d352 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-03-21 11:56:07,422 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8041406d0a1ddc5b9409edb79457d352, store=A 2023-03-21 11:56:07,422 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:56:07,422 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8041406d0a1ddc5b9409edb79457d352, store=B 2023-03-21 11:56:07,422 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:56:07,422 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8041406d0a1ddc5b9409edb79457d352, store=C 2023-03-21 11:56:07,422 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:56:07,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(9158): Flush requested on 8041406d0a1ddc5b9409edb79457d352 2023-03-21 11:56:07,425 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. as already flushing 2023-03-21 11:56:07,468 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679399767449, max timestamp=1679399767463Num uniques 859; 2023-03-21 11:56:07,468 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679399767449, max timestamp=1679399767463Num uniques 859; 2023-03-21 11:56:07,468 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679399767449, max timestamp=1679399767463Num uniques 859; 2023-03-21 11:56:07,468 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-21 11:56:07,468 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-21 11:56:07,468 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-21 11:56:07,495 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:07,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55208 deadline: 1679399827490, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:07,496 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:07,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55188 deadline: 1679399827491, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:07,496 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:07,496 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:07,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55190 deadline: 1679399827494, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:07,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55216 deadline: 1679399827495, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:07,496 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:07,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55180 deadline: 1679399827492, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:07,534 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230321b3f419f2c86e4ae58a8bfef5c3e857ff to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230321b3f419f2c86e4ae58a8bfef5c3e857ff 2023-03-21 11:56:07,553 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=13, memsize=13.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/A/b1457dfd3fc0408281a28e09e36d2169 2023-03-21 11:56:07,602 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:07,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55188 deadline: 1679399827599, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:07,604 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:07,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55208 deadline: 1679399827599, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:07,604 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:07,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55180 deadline: 1679399827603, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:07,604 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:07,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55216 deadline: 1679399827603, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:07,604 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:07,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55190 deadline: 1679399827603, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:07,808 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:07,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55190 deadline: 1679399827806, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:07,809 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:07,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55208 deadline: 1679399827806, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:07,809 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:07,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55216 deadline: 1679399827807, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:07,813 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:07,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55180 deadline: 1679399827807, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:07,814 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:07,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55188 deadline: 1679399827807, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:08,029 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=13 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/B/40ba77cc4ae04c7495923c06f5df12d9 2023-03-21 11:56:08,050 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=13 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/C/f3cdb56e0bdb4a9bbb57038bfd3f1f8b 2023-03-21 11:56:08,057 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/A/b1457dfd3fc0408281a28e09e36d2169 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/b1457dfd3fc0408281a28e09e36d2169 2023-03-21 11:56:08,064 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/b1457dfd3fc0408281a28e09e36d2169, entries=150, sequenceid=13, filesize=25.1 K 2023-03-21 11:56:08,065 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/B/40ba77cc4ae04c7495923c06f5df12d9 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/40ba77cc4ae04c7495923c06f5df12d9 2023-03-21 11:56:08,098 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/40ba77cc4ae04c7495923c06f5df12d9, entries=150, sequenceid=13, filesize=11.6 K 2023-03-21 11:56:08,101 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/C/f3cdb56e0bdb4a9bbb57038bfd3f1f8b as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/f3cdb56e0bdb4a9bbb57038bfd3f1f8b 2023-03-21 11:56:08,108 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/f3cdb56e0bdb4a9bbb57038bfd3f1f8b, entries=150, sequenceid=13, filesize=11.6 K 2023-03-21 11:56:08,109 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=140.89 KB/144270 for 8041406d0a1ddc5b9409edb79457d352 in 697ms, sequenceid=13, compaction requested=false 2023-03-21 11:56:08,109 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-03-21 11:56:08,112 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8041406d0a1ddc5b9409edb79457d352: 2023-03-21 11:56:08,112 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. 2023-03-21 11:56:08,112 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:56:08,112 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:56:08,113 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:08,114 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:56:08,114 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:56:08,116 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:08,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:08,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:08,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:08,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:08,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:08,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:08,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:08,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:08,118 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:56:08,118 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ddc7afb[Count = 0] remaining members to acquire global barrier 2023-03-21 11:56:08,118 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:56:08,118 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:08,119 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:08,119 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:56:08,122 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:08,123 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(162): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:08,123 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:08,123 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:56:08,123 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:56:08,123 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:56:08,123 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:56:08,125 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:56:08,125 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:08,125 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:56:08,125 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:08,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:08,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:08,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:08,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:08,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(9158): Flush requested on 8041406d0a1ddc5b9409edb79457d352 2023-03-21 11:56:08,126 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8041406d0a1ddc5b9409edb79457d352 3/3 column families, dataSize=154.31 KB heapSize=440.77 KB 2023-03-21 11:56:08,128 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1679399767474, max timestamp=1679399768124Num uniques 1068; 2023-03-21 11:56:08,128 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-03-21 11:56:08,128 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1679399767474, max timestamp=1679399768124Num uniques 1068; 2023-03-21 11:56:08,128 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8041406d0a1ddc5b9409edb79457d352, store=A 2023-03-21 11:56:08,128 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-03-21 11:56:08,128 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-21 11:56:08,128 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8041406d0a1ddc5b9409edb79457d352, store=B 2023-03-21 11:56:08,128 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-21 11:56:08,128 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8041406d0a1ddc5b9409edb79457d352, store=C 2023-03-21 11:56:08,128 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-21 11:56:08,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:08,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:08,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:08,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:08,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:08,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:08,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:56:08,132 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:56:08,132 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:56:08,132 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:56:08,132 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:56:08,132 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:56:08,132 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:56:08,147 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:08,147 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:08,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:08,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:08,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:08,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:08,148 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:08,147 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:08,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:08,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:08,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:08,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:08,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:08,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:08,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:08,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:08,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:08,149 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:08,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:08,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:08,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:08,154 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:08,154 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:56:08,154 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:56:08,155 DEBUG [Thread-776] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:56:08,155 DEBUG [Thread-776] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:56:08,154 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:08,154 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:08,154 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:08,155 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:08,155 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:08,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:08,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:08,155 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:08,155 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:08,155 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:08,155 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:08,155 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:08,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:08,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:08,178 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230321ac91c469b2c944f4b211726ae2257203 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230321ac91c469b2c944f4b211726ae2257203 2023-03-21 11:56:08,207 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:08,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55180 deadline: 1679399828204, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:08,207 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:08,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55216 deadline: 1679399828206, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:08,208 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:08,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55190 deadline: 1679399828206, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:08,211 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:08,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55188 deadline: 1679399828208, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:08,229 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:08,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55208 deadline: 1679399828227, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:08,255 DEBUG [Thread-776] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:56:08,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:56:08,257 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:56:08,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:56:08,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:56:08,296 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:56:08,297 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:08,297 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:56:08,297 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:56:08,297 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:08,297 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:08,298 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:08,298 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:08,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:08,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:08,298 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:08,298 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:56:08,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:08,299 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:08,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:56:08,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:08,299 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:56:08,306 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:56:08,306 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:56:08,306 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:56:08,306 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:08,306 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:56:08,307 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:56:08,307 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:56:08,307 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. 2023-03-21 11:56:08,307 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. started... 2023-03-21 11:56:08,307 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. as already flushing 2023-03-21 11:56:08,307 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. 2023-03-21 11:56:08,307 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:56:08,307 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:56:08,307 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:08,307 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:56:08,307 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:56:08,310 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:08,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55180 deadline: 1679399828309, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:08,310 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:08,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55216 deadline: 1679399828309, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:08,311 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:08,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55190 deadline: 1679399828309, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:08,312 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:08,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:08,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:08,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:08,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:08,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:08,313 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:08,313 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:08,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55188 deadline: 1679399828313, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:08,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:08,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:08,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:08,315 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:56:08,315 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3aa863a3[Count = 0] remaining members to acquire global barrier 2023-03-21 11:56:08,315 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:56:08,315 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:08,315 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:08,315 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:56:08,322 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:08,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:08,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:08,322 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:56:08,322 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:56:08,322 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:56:08,322 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:08,323 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:56:08,329 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:56:08,329 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:08,329 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:08,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:08,329 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:56:08,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:08,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:08,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:08,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:08,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:08,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:08,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:08,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:08,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:08,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:56:08,332 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:56:08,332 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:56:08,332 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:56:08,332 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:56:08,332 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:56:08,332 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:56:08,333 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:08,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55208 deadline: 1679399828331, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:08,339 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:08,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:08,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:08,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:08,339 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:08,339 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:08,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:08,339 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:08,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:08,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:08,341 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:08,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:08,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:08,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:08,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:08,345 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:08,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:08,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:08,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:08,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:08,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:08,348 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:08,348 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:08,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:08,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:08,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:08,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:08,348 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:08,348 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:08,348 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:08,349 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:08,349 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:08,349 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:08,349 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:08,349 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:08,349 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:56:08,349 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:56:08,349 DEBUG [Thread-776] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:56:08,349 DEBUG [Thread-776] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:56:08,349 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:56:08,449 DEBUG [Thread-776] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:56:08,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:56:08,452 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:56:08,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:56:08,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:56:08,474 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:56:08,474 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:08,474 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:56:08,474 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:56:08,475 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:08,475 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:08,475 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:08,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:08,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:08,476 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:08,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:08,476 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:08,476 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:56:08,476 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:08,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:56:08,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:08,477 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:56:08,480 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:56:08,480 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:56:08,480 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:56:08,480 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:08,480 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:56:08,491 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:56:08,494 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:56:08,494 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. 2023-03-21 11:56:08,494 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. started... 2023-03-21 11:56:08,494 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. as already flushing 2023-03-21 11:56:08,494 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. 2023-03-21 11:56:08,494 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:56:08,494 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:56:08,494 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:08,494 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:56:08,494 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:56:08,495 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:08,496 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:08,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:08,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:08,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:08,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:08,496 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:08,496 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:56:08,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:08,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:08,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:08,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:08,498 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:56:08,498 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@683c9f8c[Count = 0] remaining members to acquire global barrier 2023-03-21 11:56:08,498 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:56:08,498 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:08,499 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:08,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:08,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:08,499 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:56:08,499 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:56:08,499 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:56:08,499 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:08,499 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:56:08,501 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:56:08,501 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:08,501 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:08,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:08,501 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:56:08,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:08,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:08,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:08,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:08,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:08,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:08,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:08,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:08,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:08,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:56:08,508 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:56:08,508 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:56:08,508 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:56:08,508 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:56:08,508 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:56:08,508 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:56:08,513 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:08,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55180 deadline: 1679399828512, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:08,515 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:08,515 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:08,515 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:08,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:08,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:08,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:08,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:08,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:08,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:08,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:08,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:08,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:08,516 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:08,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:08,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:08,517 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:08,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:08,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55216 deadline: 1679399828513, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:08,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:08,518 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:08,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:08,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55190 deadline: 1679399828513, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:08,518 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:08,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:08,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:08,521 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:08,521 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:08,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:08,521 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:56:08,521 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:56:08,521 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:08,522 DEBUG [Thread-776] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:56:08,522 DEBUG [Thread-776] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:56:08,521 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:08,522 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:08,522 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:08,522 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:08,522 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:08,522 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:08,522 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:08,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:08,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:08,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:08,523 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:08,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55188 deadline: 1679399828516, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:08,538 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:08,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55208 deadline: 1679399828535, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:08,620 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=53, memsize=13.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/A/7c1af716a9f848afa158ef97aea37252 2023-03-21 11:56:08,622 DEBUG [Thread-776] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:56:08,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:56:08,626 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:56:08,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:56:08,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:56:08,632 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:56:08,632 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:08,632 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:56:08,632 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:56:08,633 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:08,633 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:08,634 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:08,634 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:08,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:08,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:08,634 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:08,634 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:56:08,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:08,634 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:08,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:56:08,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:08,635 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:56:08,638 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:56:08,638 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:56:08,638 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:56:08,638 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:08,638 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:56:08,638 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:56:08,638 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:56:08,638 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. 2023-03-21 11:56:08,638 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. started... 2023-03-21 11:56:08,639 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. as already flushing 2023-03-21 11:56:08,639 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. 2023-03-21 11:56:08,639 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:56:08,639 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:56:08,639 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:08,639 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:56:08,639 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:56:08,647 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:08,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:08,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:08,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:08,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:08,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:08,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:08,648 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:08,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:08,649 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:08,649 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:56:08,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:08,649 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:56:08,649 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@790e7aac[Count = 0] remaining members to acquire global barrier 2023-03-21 11:56:08,649 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:56:08,649 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:08,654 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:08,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:08,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:08,654 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:56:08,654 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:56:08,654 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:56:08,654 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:08,654 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:56:08,655 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:56:08,655 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:08,655 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:56:08,658 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:08,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:08,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:08,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:08,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:08,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:08,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:08,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:08,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:08,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:08,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:08,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:56:08,661 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:56:08,661 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:56:08,661 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:56:08,661 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:56:08,661 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:56:08,661 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:56:08,664 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:08,664 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:08,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:08,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:08,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:08,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:08,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:08,665 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:08,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:08,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:08,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:08,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:08,665 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:08,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:08,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:08,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:08,666 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:08,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:08,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:08,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:08,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:08,672 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:08,672 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:08,672 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:56:08,672 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:56:08,672 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:08,673 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:08,672 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:08,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:08,673 DEBUG [Thread-776] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:56:08,673 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:08,673 DEBUG [Thread-776] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:56:08,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:08,673 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:08,673 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:08,673 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:08,673 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:08,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:08,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:08,773 DEBUG [Thread-776] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:56:08,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:56:08,776 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:56:08,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:56:08,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:56:08,777 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:56:08,777 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:08,777 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:56:08,777 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:56:08,778 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:08,778 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:08,779 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:08,779 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:08,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:08,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:08,779 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:08,779 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:56:08,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:08,780 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:08,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:56:08,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:08,780 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:56:08,780 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:56:08,780 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:56:08,781 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:08,781 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:56:08,781 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:56:08,781 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:56:08,781 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. 2023-03-21 11:56:08,781 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. started... 2023-03-21 11:56:08,781 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. as already flushing 2023-03-21 11:56:08,781 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. 2023-03-21 11:56:08,781 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:56:08,781 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:56:08,781 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:08,781 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:56:08,781 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:56:08,782 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:08,782 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:08,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:08,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:08,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:08,783 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:08,783 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:56:08,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:08,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:08,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:08,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:08,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:08,784 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:56:08,784 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:56:08,784 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@27dbdcd4[Count = 0] remaining members to acquire global barrier 2023-03-21 11:56:08,784 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:08,785 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:08,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:08,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:08,785 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:56:08,785 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:56:08,785 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:56:08,786 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:08,786 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:56:08,787 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:56:08,787 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:08,787 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:56:08,787 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:08,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:08,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:08,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:08,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:08,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:08,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:08,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:08,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:08,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:08,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:08,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:56:08,789 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:56:08,789 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:56:08,789 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:56:08,789 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:56:08,789 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:56:08,789 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:56:08,790 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:08,790 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:08,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:08,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:08,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:08,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:08,790 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:08,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:08,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:08,791 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:08,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:08,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:08,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:08,792 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:08,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:08,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:08,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:08,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:08,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:08,797 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:08,797 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:08,797 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:08,797 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:08,797 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:08,797 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:08,797 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:08,798 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:08,798 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:56:08,798 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:56:08,798 DEBUG [Thread-776] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:56:08,798 DEBUG [Thread-776] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:56:08,798 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:08,798 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:08,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:08,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:08,798 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:56:08,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:08,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:08,816 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:08,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55180 deadline: 1679399828815, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:08,821 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:08,824 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:08,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55216 deadline: 1679399828821, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:08,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55190 deadline: 1679399828822, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:08,829 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:08,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55188 deadline: 1679399828825, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:08,844 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:08,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55208 deadline: 1679399828842, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:08,899 DEBUG [Thread-776] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:56:08,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:56:08,901 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:56:08,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:56:08,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:56:08,905 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:56:08,905 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:08,905 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:56:08,905 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:56:08,905 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:08,905 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:08,906 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:08,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:08,906 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:08,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:08,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:08,907 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:08,907 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:56:08,907 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:08,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:56:08,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:08,907 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:56:08,910 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:56:08,910 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:56:08,910 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:56:08,910 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:08,910 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:56:08,911 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:56:08,911 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:56:08,911 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. 2023-03-21 11:56:08,911 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. started... 2023-03-21 11:56:08,911 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. as already flushing 2023-03-21 11:56:08,911 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. 2023-03-21 11:56:08,911 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:56:08,911 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:56:08,911 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:08,911 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:56:08,911 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:56:08,921 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:08,921 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:08,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:08,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:08,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:08,921 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:08,921 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:56:08,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:08,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:08,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:08,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:08,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:08,923 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:56:08,923 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:56:08,923 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e1706f7[Count = 0] remaining members to acquire global barrier 2023-03-21 11:56:08,923 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:08,924 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:08,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:08,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:08,924 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:56:08,924 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:56:08,924 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:08,924 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:56:08,924 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:56:08,925 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:56:08,925 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:08,925 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:08,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:08,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:08,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:08,925 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:56:08,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:08,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:08,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:08,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:08,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:08,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:08,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:08,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:56:08,927 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:56:08,927 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:56:08,927 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:56:08,927 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:56:08,927 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:56:08,927 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:56:08,928 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:08,928 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:08,928 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:08,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:08,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:08,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:08,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:08,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:08,929 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:08,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:08,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:08,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:08,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:08,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:08,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:08,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:08,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:08,930 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:08,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:08,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:08,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:08,932 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:08,932 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:08,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:08,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:08,933 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:08,933 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:08,933 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:08,933 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:08,933 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:08,933 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:08,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:08,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:08,933 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:56:08,933 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:56:08,933 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:08,934 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:08,934 DEBUG [Thread-776] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:56:08,934 DEBUG [Thread-776] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:56:09,034 DEBUG [Thread-776] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:56:09,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:56:09,039 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:56:09,041 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=53 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/B/6d4b7ace0bd54798b215a54de28b75cb 2023-03-21 11:56:09,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:56:09,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:56:09,049 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:56:09,049 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:09,049 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:56:09,049 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:56:09,050 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:09,050 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:09,050 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:09,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:09,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:09,051 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:09,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:09,051 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:09,051 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:56:09,051 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:09,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:56:09,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:09,052 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:56:09,052 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:56:09,052 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:56:09,052 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:56:09,052 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:09,052 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:56:09,052 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:56:09,052 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:56:09,052 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. 2023-03-21 11:56:09,052 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. started... 2023-03-21 11:56:09,052 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. as already flushing 2023-03-21 11:56:09,053 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. 2023-03-21 11:56:09,053 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:56:09,053 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:56:09,053 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:09,053 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:56:09,053 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:56:09,057 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:09,057 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:09,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:09,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:09,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:09,057 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:09,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:09,058 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:56:09,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:09,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:09,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:09,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:09,063 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:56:09,063 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@525b2246[Count = 0] remaining members to acquire global barrier 2023-03-21 11:56:09,063 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:56:09,063 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:09,072 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:09,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:09,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:09,072 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:09,072 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:56:09,072 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:56:09,072 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:56:09,072 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:56:09,079 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:09,079 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:56:09,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:09,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:09,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:09,079 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:09,079 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:56:09,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:09,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:09,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:09,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:09,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:09,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:09,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:09,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:56:09,081 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:56:09,081 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:56:09,081 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:56:09,081 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:56:09,081 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:56:09,081 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:56:09,089 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:09,089 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:09,089 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:09,089 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:09,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:09,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:09,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:09,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:09,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:09,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:09,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:09,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:09,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:09,091 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:09,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:09,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:09,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:09,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:09,096 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:09,096 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:09,096 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:56:09,096 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:56:09,096 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:09,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:09,096 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:09,096 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:09,096 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:09,096 DEBUG [Thread-776] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:56:09,096 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:09,096 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:09,096 DEBUG [Thread-776] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:56:09,096 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:56:09,098 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:09,098 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:09,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:09,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:09,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:09,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:09,196 DEBUG [Thread-776] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:56:09,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:56:09,199 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:56:09,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:56:09,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:56:09,201 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:56:09,201 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:09,201 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:56:09,201 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:56:09,201 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:09,201 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:09,202 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:09,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:09,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:09,202 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:09,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:09,203 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:09,203 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:56:09,203 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:09,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:56:09,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:09,203 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:56:09,204 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:56:09,204 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:56:09,204 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:56:09,204 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:09,204 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:56:09,205 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:56:09,205 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:56:09,205 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. 2023-03-21 11:56:09,205 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. started... 2023-03-21 11:56:09,205 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. as already flushing 2023-03-21 11:56:09,205 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. 2023-03-21 11:56:09,205 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:56:09,205 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:56:09,205 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:09,205 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:56:09,205 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:56:09,206 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:09,207 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:09,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:09,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:09,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:09,207 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:09,207 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:56:09,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:09,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:09,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:09,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:09,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:09,211 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:56:09,211 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@588ae9bb[Count = 0] remaining members to acquire global barrier 2023-03-21 11:56:09,211 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:56:09,211 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:09,212 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:09,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:09,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:09,212 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:56:09,212 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:56:09,212 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:56:09,212 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:09,212 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:56:09,213 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:56:09,213 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:09,213 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:09,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:09,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:09,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:09,213 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:56:09,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:09,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:09,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:09,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:09,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:09,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:09,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:09,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:56:09,215 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:56:09,215 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:56:09,215 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:56:09,215 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:56:09,215 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:56:09,215 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:56:09,216 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:09,216 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:09,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:09,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:09,216 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:09,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:09,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:09,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:09,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:09,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:09,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:09,217 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:09,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:09,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:09,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:09,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:09,219 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:09,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:09,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:09,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:09,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:09,221 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:09,221 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:09,221 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:09,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:09,221 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:56:09,221 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:56:09,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:09,221 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:09,221 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:09,221 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:09,221 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:09,221 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:09,221 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:09,221 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:09,221 DEBUG [Thread-776] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:56:09,221 DEBUG [Thread-776] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:56:09,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:09,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:09,256 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-03-21 11:56:09,321 DEBUG [Thread-776] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:56:09,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:56:09,322 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:09,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55180 deadline: 1679399829319, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:09,328 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:56:09,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:56:09,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:56:09,329 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:56:09,329 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:09,329 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:56:09,329 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:56:09,330 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:09,330 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:09,330 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:09,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55190 deadline: 1679399829330, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:09,330 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:09,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:09,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:09,331 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:09,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:09,331 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:09,331 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:56:09,331 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:09,331 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:09,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55216 deadline: 1679399829331, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:09,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:56:09,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:09,332 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:56:09,332 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:56:09,332 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:56:09,333 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:56:09,333 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:09,333 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:56:09,335 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:09,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55188 deadline: 1679399829333, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:09,346 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:56:09,346 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:56:09,349 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:09,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55208 deadline: 1679399829348, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:09,355 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. 2023-03-21 11:56:09,355 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. started... 2023-03-21 11:56:09,355 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. as already flushing 2023-03-21 11:56:09,355 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. 2023-03-21 11:56:09,355 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:56:09,355 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:56:09,355 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:09,355 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:56:09,355 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:56:09,357 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:09,357 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:09,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:09,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:09,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:09,358 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:09,358 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:56:09,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:09,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:09,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:09,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:09,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:09,359 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:56:09,359 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@21cea302[Count = 0] remaining members to acquire global barrier 2023-03-21 11:56:09,359 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:56:09,359 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:09,360 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:09,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:09,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:09,360 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:56:09,360 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:56:09,360 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:56:09,361 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:09,361 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:56:09,361 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:09,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:09,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:09,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:09,361 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:56:09,361 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:09,361 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:56:09,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:09,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:09,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:09,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:09,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:09,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:09,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:09,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:56:09,364 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:56:09,364 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:56:09,364 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:56:09,364 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:56:09,364 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:56:09,364 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:56:09,365 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:09,365 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:09,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:09,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:09,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:09,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:09,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:09,367 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:09,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:09,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:09,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:09,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:09,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:09,368 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:09,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:09,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:09,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:09,369 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:09,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:09,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:09,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:09,374 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:09,374 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:09,374 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:09,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:09,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:09,374 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:09,374 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:56:09,374 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:56:09,375 DEBUG [Thread-776] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:56:09,375 DEBUG [Thread-776] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:56:09,374 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:09,375 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:09,375 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:09,375 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:09,375 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:09,375 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:09,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:09,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:09,461 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=53 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/C/c144e9fdcfbc498bb26bce99c8df5d7c 2023-03-21 11:56:09,468 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/A/7c1af716a9f848afa158ef97aea37252 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/7c1af716a9f848afa158ef97aea37252 2023-03-21 11:56:09,475 DEBUG [Thread-776] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:56:09,475 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/7c1af716a9f848afa158ef97aea37252, entries=150, sequenceid=53, filesize=25.1 K 2023-03-21 11:56:09,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:56:09,478 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:56:09,479 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/B/6d4b7ace0bd54798b215a54de28b75cb as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/6d4b7ace0bd54798b215a54de28b75cb 2023-03-21 11:56:09,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:56:09,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:56:09,490 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:56:09,490 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:09,490 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:56:09,490 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:56:09,491 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:09,491 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:09,491 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:09,492 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:09,492 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:56:09,492 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:09,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:09,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:09,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:09,492 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:09,492 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/6d4b7ace0bd54798b215a54de28b75cb, entries=150, sequenceid=53, filesize=11.6 K 2023-03-21 11:56:09,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:56:09,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:09,493 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:56:09,493 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/C/c144e9fdcfbc498bb26bce99c8df5d7c as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/c144e9fdcfbc498bb26bce99c8df5d7c 2023-03-21 11:56:09,494 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:56:09,494 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:56:09,494 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:56:09,494 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:09,494 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:56:09,497 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:56:09,497 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:56:09,497 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. 2023-03-21 11:56:09,497 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. started... 2023-03-21 11:56:09,497 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. as already flushing 2023-03-21 11:56:09,497 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. 2023-03-21 11:56:09,497 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:56:09,497 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:56:09,497 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:09,497 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:56:09,497 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:56:09,498 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/c144e9fdcfbc498bb26bce99c8df5d7c, entries=150, sequenceid=53, filesize=11.6 K 2023-03-21 11:56:09,502 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~223.67 KB/229040, currentSize=107.34 KB/109920 for 8041406d0a1ddc5b9409edb79457d352 in 1376ms, sequenceid=53, compaction requested=false 2023-03-21 11:56:09,502 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8041406d0a1ddc5b9409edb79457d352: 2023-03-21 11:56:09,514 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:09,514 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:09,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:09,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:09,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:09,514 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:09,514 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:56:09,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:09,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:09,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:09,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:09,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:09,516 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:56:09,516 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@47883972[Count = 0] remaining members to acquire global barrier 2023-03-21 11:56:09,516 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:56:09,516 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:09,516 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:09,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:09,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:09,517 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:56:09,517 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:56:09,517 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:56:09,517 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:09,517 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:56:09,518 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:09,518 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:56:09,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:09,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:09,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:09,518 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:09,518 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:56:09,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:09,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:09,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:09,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:09,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:09,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:09,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:09,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:56:09,520 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:56:09,520 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:56:09,520 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:56:09,520 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:56:09,520 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:56:09,520 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:56:09,521 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:09,521 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:09,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:09,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:09,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:09,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:09,522 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:09,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:09,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:09,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:09,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:09,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:09,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:09,523 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:09,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:09,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:09,524 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:09,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:09,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:09,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:09,529 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:09,529 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:09,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:09,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:09,529 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:09,529 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:09,529 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:09,529 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:09,529 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:09,529 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:09,529 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:09,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:09,530 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:56:09,530 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:56:09,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:09,529 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:09,530 DEBUG [Thread-776] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:56:09,530 DEBUG [Thread-776] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:56:09,531 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:56:09,630 DEBUG [Thread-776] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:56:09,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:56:09,632 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:56:09,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:56:09,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:56:09,634 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:56:09,634 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:09,634 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:56:09,634 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:56:09,634 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:09,634 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:09,635 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:09,635 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:09,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:09,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:09,636 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:09,636 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:56:09,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:09,636 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:09,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:56:09,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:09,636 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:56:09,637 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:56:09,637 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:56:09,637 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:56:09,637 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:09,637 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:56:09,637 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:56:09,638 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:56:09,638 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. 2023-03-21 11:56:09,638 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. started... 2023-03-21 11:56:09,638 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8041406d0a1ddc5b9409edb79457d352 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-21 11:56:09,638 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8041406d0a1ddc5b9409edb79457d352, store=A 2023-03-21 11:56:09,638 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:56:09,638 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8041406d0a1ddc5b9409edb79457d352, store=B 2023-03-21 11:56:09,638 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:56:09,638 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8041406d0a1ddc5b9409edb79457d352, store=C 2023-03-21 11:56:09,638 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:56:10,076 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032140c1aeb96b1a4403a55fcf5fba44bf72 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032140c1aeb96b1a4403a55fcf5fba44bf72 2023-03-21 11:56:10,096 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=72, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/A/27e21010df214d5c8ea61b10d07a0e7c 2023-03-21 11:56:10,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(9158): Flush requested on 8041406d0a1ddc5b9409edb79457d352 2023-03-21 11:56:10,336 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. as already flushing 2023-03-21 11:56:10,352 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:10,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55188 deadline: 1679399830347, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:10,353 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:10,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55190 deadline: 1679399830348, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:10,353 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:10,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55180 deadline: 1679399830348, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:10,353 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:10,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55216 deadline: 1679399830348, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:10,362 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:10,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55208 deadline: 1679399830360, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:10,457 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:10,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55188 deadline: 1679399830454, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:10,457 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:10,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55190 deadline: 1679399830455, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:10,457 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:10,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55180 deadline: 1679399830455, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:10,458 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:10,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55216 deadline: 1679399830455, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:10,515 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=72 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/B/05d3c84401744612ba8ac5a5563e06cd 2023-03-21 11:56:10,658 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:10,658 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:10,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55188 deadline: 1679399830658, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:10,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55190 deadline: 1679399830658, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:10,660 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:10,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55180 deadline: 1679399830660, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:10,665 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:10,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55216 deadline: 1679399830663, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:10,943 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=72 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/C/93e36abfb16c4fd88667a1278a8b2e5a 2023-03-21 11:56:10,948 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/A/27e21010df214d5c8ea61b10d07a0e7c as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/27e21010df214d5c8ea61b10d07a0e7c 2023-03-21 11:56:10,952 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/27e21010df214d5c8ea61b10d07a0e7c, entries=150, sequenceid=72, filesize=25.1 K 2023-03-21 11:56:10,953 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/B/05d3c84401744612ba8ac5a5563e06cd as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/05d3c84401744612ba8ac5a5563e06cd 2023-03-21 11:56:10,957 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/05d3c84401744612ba8ac5a5563e06cd, entries=150, sequenceid=72, filesize=11.6 K 2023-03-21 11:56:10,960 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/C/93e36abfb16c4fd88667a1278a8b2e5a as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/93e36abfb16c4fd88667a1278a8b2e5a 2023-03-21 11:56:10,976 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:10,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55180 deadline: 1679399830967, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:10,976 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:10,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55188 deadline: 1679399830967, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:10,977 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:10,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55190 deadline: 1679399830967, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:10,977 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:10,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55216 deadline: 1679399830968, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:10,978 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/93e36abfb16c4fd88667a1278a8b2e5a, entries=150, sequenceid=72, filesize=11.6 K 2023-03-21 11:56:10,979 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 8041406d0a1ddc5b9409edb79457d352 in 1341ms, sequenceid=72, compaction requested=true 2023-03-21 11:56:10,979 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8041406d0a1ddc5b9409edb79457d352: 2023-03-21 11:56:10,979 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. 2023-03-21 11:56:10,980 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:56:10,980 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:56:10,980 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:10,980 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:56:10,980 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:56:10,981 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:10,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:10,981 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:10,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:10,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:10,981 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:10,981 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:56:10,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:10,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:10,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:10,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:10,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:10,986 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:56:10,986 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c4a6ad7[Count = 0] remaining members to acquire global barrier 2023-03-21 11:56:10,986 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:56:10,986 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:10,987 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:10,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:10,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:10,987 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:56:10,987 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:10,987 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:56:10,987 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:56:10,987 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:56:10,988 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:10,988 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:56:10,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:10,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:10,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:10,988 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:10,988 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:56:10,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:10,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:10,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:10,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:10,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:10,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:10,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:10,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:56:10,990 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:56:10,990 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:56:10,991 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:56:10,991 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:56:10,991 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:56:10,991 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:56:10,992 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:10,992 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:10,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:10,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:10,992 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:10,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:10,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:10,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:10,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:10,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:10,993 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:10,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:10,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:10,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:10,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:10,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:10,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:10,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:10,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:10,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:10,996 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:11,005 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:11,006 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:11,006 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:11,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:11,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:11,006 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:11,006 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:56:11,006 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:56:11,006 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:11,006 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:11,006 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:11,006 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:11,006 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:11,006 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:11,006 DEBUG [Thread-776] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:56:11,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:11,006 DEBUG [Thread-776] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:56:11,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:11,106 DEBUG [Thread-776] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:56:11,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:56:11,108 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:56:11,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:56:11,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:56:11,109 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:56:11,109 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:11,109 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:56:11,109 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:56:11,110 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:11,110 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:11,111 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:11,111 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:11,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:11,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:11,111 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:11,111 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:56:11,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:11,111 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:11,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:56:11,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:11,112 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:56:11,112 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:56:11,112 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:56:11,112 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:56:11,112 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:11,112 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:56:11,112 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:56:11,112 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:56:11,112 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. 2023-03-21 11:56:11,112 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. started... 2023-03-21 11:56:11,112 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8041406d0a1ddc5b9409edb79457d352 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-21 11:56:11,112 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8041406d0a1ddc5b9409edb79457d352, store=A 2023-03-21 11:56:11,113 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:56:11,113 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8041406d0a1ddc5b9409edb79457d352, store=B 2023-03-21 11:56:11,113 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:56:11,113 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8041406d0a1ddc5b9409edb79457d352, store=C 2023-03-21 11:56:11,113 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:56:11,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(9158): Flush requested on 8041406d0a1ddc5b9409edb79457d352 2023-03-21 11:56:11,483 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. as already flushing 2023-03-21 11:56:11,521 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:11,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55190 deadline: 1679399831516, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:11,524 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:11,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55216 deadline: 1679399831521, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:11,524 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:11,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55188 deadline: 1679399831521, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:11,524 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:11,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55180 deadline: 1679399831522, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:11,535 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230321ba86542d6f6347588f35c93914c07b73 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230321ba86542d6f6347588f35c93914c07b73 2023-03-21 11:56:11,545 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=85, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/A/47c811f81290442490e545740b1bd5a1 2023-03-21 11:56:11,576 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=85 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/B/a47ca18fe7904b6a9d572f9a8bacdaed 2023-03-21 11:56:11,614 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=85 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/C/77dc6fa1beac42d4ae2ca72b468e18dc 2023-03-21 11:56:11,635 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:11,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55216 deadline: 1679399831635, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:11,636 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:11,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55188 deadline: 1679399831635, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:11,639 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:11,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55190 deadline: 1679399831635, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:11,640 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:11,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55180 deadline: 1679399831636, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:11,642 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/A/47c811f81290442490e545740b1bd5a1 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/47c811f81290442490e545740b1bd5a1 2023-03-21 11:56:11,647 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/47c811f81290442490e545740b1bd5a1, entries=100, sequenceid=85, filesize=18.4 K 2023-03-21 11:56:11,649 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/B/a47ca18fe7904b6a9d572f9a8bacdaed as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/a47ca18fe7904b6a9d572f9a8bacdaed 2023-03-21 11:56:11,654 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/a47ca18fe7904b6a9d572f9a8bacdaed, entries=100, sequenceid=85, filesize=9.3 K 2023-03-21 11:56:11,655 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/C/77dc6fa1beac42d4ae2ca72b468e18dc as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/77dc6fa1beac42d4ae2ca72b468e18dc 2023-03-21 11:56:11,660 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/77dc6fa1beac42d4ae2ca72b468e18dc, entries=100, sequenceid=85, filesize=9.3 K 2023-03-21 11:56:11,661 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=114.05 KB/116790 for 8041406d0a1ddc5b9409edb79457d352 in 548ms, sequenceid=85, compaction requested=true 2023-03-21 11:56:11,661 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8041406d0a1ddc5b9409edb79457d352: 2023-03-21 11:56:11,661 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. 2023-03-21 11:56:11,661 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:56:11,661 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:56:11,661 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:11,661 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:56:11,661 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:56:11,663 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:11,663 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:11,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:11,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:11,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:11,663 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:11,663 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:56:11,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:11,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:11,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:11,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:11,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:11,664 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:56:11,664 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@438285d4[Count = 0] remaining members to acquire global barrier 2023-03-21 11:56:11,664 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:56:11,664 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:11,665 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:11,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:11,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:11,665 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:56:11,665 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:56:11,665 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:11,665 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:56:11,665 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:56:11,666 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:11,666 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:56:11,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:11,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:11,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:11,666 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:11,666 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:56:11,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:11,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:11,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:11,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:11,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:11,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:11,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:11,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:56:11,668 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:56:11,668 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:56:11,668 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:56:11,668 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:56:11,668 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:56:11,668 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:56:11,669 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:11,669 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:11,669 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:11,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:11,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:11,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:11,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:11,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:11,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:11,669 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:11,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:11,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:11,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:11,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:11,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:11,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:11,670 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:11,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:11,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:11,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:11,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:11,673 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:11,673 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:11,673 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:11,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:11,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:11,673 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:56:11,673 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:56:11,674 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:56:11,673 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:11,673 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:11,674 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:11,674 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:11,674 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:11,674 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:11,674 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:11,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:11,674 DEBUG [Thread-776] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:56:11,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:11,674 DEBUG [Thread-776] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:56:11,774 DEBUG [Thread-776] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:56:11,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:56:11,777 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:56:11,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:56:11,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:56:11,778 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:56:11,778 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:11,778 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:56:11,778 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:56:11,778 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:11,778 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:11,779 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:11,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:11,779 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:11,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:11,779 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:11,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:11,779 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:56:11,779 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:11,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:56:11,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:11,780 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:56:11,780 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:56:11,780 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:56:11,780 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:56:11,780 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:11,780 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:56:11,780 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:56:11,780 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. 2023-03-21 11:56:11,780 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. started... 2023-03-21 11:56:11,780 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:56:11,781 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8041406d0a1ddc5b9409edb79457d352 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-21 11:56:11,781 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8041406d0a1ddc5b9409edb79457d352, store=A 2023-03-21 11:56:11,781 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:56:11,781 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8041406d0a1ddc5b9409edb79457d352, store=B 2023-03-21 11:56:11,781 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:56:11,781 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8041406d0a1ddc5b9409edb79457d352, store=C 2023-03-21 11:56:11,781 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:56:11,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(9158): Flush requested on 8041406d0a1ddc5b9409edb79457d352 2023-03-21 11:56:11,839 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. as already flushing 2023-03-21 11:56:11,851 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:11,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55188 deadline: 1679399831848, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:11,852 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:11,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55216 deadline: 1679399831849, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:11,852 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:11,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55180 deadline: 1679399831849, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:11,853 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:11,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55190 deadline: 1679399831849, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:11,954 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:11,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55188 deadline: 1679399831952, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:11,954 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:11,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55216 deadline: 1679399831953, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:11,954 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:11,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55180 deadline: 1679399831953, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:11,955 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:11,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55190 deadline: 1679399831954, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:12,158 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:12,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55216 deadline: 1679399832155, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:12,158 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:12,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55188 deadline: 1679399832155, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:12,158 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:12,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55180 deadline: 1679399832155, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:12,158 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:12,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55190 deadline: 1679399832155, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:12,207 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230321280b63f80cd44ae3a92093708b6b92a8 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230321280b63f80cd44ae3a92093708b6b92a8 2023-03-21 11:56:12,384 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:12,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55208 deadline: 1679399832383, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:12,387 DEBUG [Thread-766] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4160 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352., hostname=jenkins-hbase20.apache.org,34369,1679399732030, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-21 11:56:12,461 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:12,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55188 deadline: 1679399832459, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:12,462 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:12,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55180 deadline: 1679399832461, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:12,470 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:12,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55190 deadline: 1679399832466, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:12,471 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:12,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55216 deadline: 1679399832466, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:12,615 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=105, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/A/83e40448e5a6453bb520cb8813d3fb16 2023-03-21 11:56:12,626 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=105 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/B/eb3995f2026d4b9b8790df5a88aa495f 2023-03-21 11:56:12,967 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:12,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55188 deadline: 1679399832964, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:12,967 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:12,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55180 deadline: 1679399832966, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:12,975 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:12,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55190 deadline: 1679399832975, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:12,979 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:12,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55216 deadline: 1679399832976, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:13,044 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=105 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/C/6264dd4f5b884048bdf3868360a8bd7f 2023-03-21 11:56:13,049 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/A/83e40448e5a6453bb520cb8813d3fb16 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/83e40448e5a6453bb520cb8813d3fb16 2023-03-21 11:56:13,053 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/83e40448e5a6453bb520cb8813d3fb16, entries=150, sequenceid=105, filesize=25.1 K 2023-03-21 11:56:13,054 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/B/eb3995f2026d4b9b8790df5a88aa495f as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/eb3995f2026d4b9b8790df5a88aa495f 2023-03-21 11:56:13,061 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/eb3995f2026d4b9b8790df5a88aa495f, entries=150, sequenceid=105, filesize=11.6 K 2023-03-21 11:56:13,062 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/C/6264dd4f5b884048bdf3868360a8bd7f as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/6264dd4f5b884048bdf3868360a8bd7f 2023-03-21 11:56:13,067 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/6264dd4f5b884048bdf3868360a8bd7f, entries=150, sequenceid=105, filesize=11.6 K 2023-03-21 11:56:13,068 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=67.09 KB/68700 for 8041406d0a1ddc5b9409edb79457d352 in 1288ms, sequenceid=105, compaction requested=true 2023-03-21 11:56:13,068 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8041406d0a1ddc5b9409edb79457d352: 2023-03-21 11:56:13,068 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. 2023-03-21 11:56:13,068 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:56:13,068 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:56:13,068 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:13,070 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:56:13,070 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:56:13,071 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:13,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:13,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:13,071 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:13,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:13,071 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:13,071 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:56:13,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:13,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:13,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:13,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:13,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:13,072 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:56:13,072 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1700f815[Count = 0] remaining members to acquire global barrier 2023-03-21 11:56:13,072 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:56:13,072 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:13,073 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:13,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:13,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:13,073 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:56:13,073 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:13,073 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:56:13,073 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:56:13,073 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:56:13,074 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:56:13,074 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:13,074 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:13,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:13,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:13,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:13,074 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:56:13,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:13,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:13,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:13,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:13,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:13,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:13,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:13,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:56:13,075 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:56:13,075 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:56:13,076 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:56:13,076 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:56:13,076 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:56:13,076 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:56:13,076 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:13,077 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:13,077 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:13,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:13,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:13,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:13,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:13,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:13,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:13,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:13,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:13,077 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:13,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:13,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:13,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:13,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:13,091 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:13,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:13,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:13,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:13,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:13,093 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:13,093 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:13,093 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:13,093 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:13,093 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:13,093 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:13,093 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:13,093 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:13,093 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:56:13,093 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:56:13,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:13,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:13,094 DEBUG [Thread-776] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:56:13,094 DEBUG [Thread-776] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:56:13,093 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:13,093 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:13,094 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:56:13,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:13,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:13,194 DEBUG [Thread-776] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:56:13,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:56:13,196 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:56:13,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:56:13,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:56:13,197 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:56:13,198 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:13,198 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:56:13,198 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:56:13,198 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:13,198 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:13,199 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:13,199 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:13,199 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:13,199 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:56:13,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:13,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:13,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:13,200 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:13,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:56:13,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:13,201 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:56:13,201 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:56:13,201 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:56:13,201 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:56:13,201 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:13,201 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:56:13,201 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:56:13,201 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:56:13,201 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. 2023-03-21 11:56:13,201 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. started... 2023-03-21 11:56:13,202 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8041406d0a1ddc5b9409edb79457d352 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-21 11:56:13,202 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8041406d0a1ddc5b9409edb79457d352, store=A 2023-03-21 11:56:13,202 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:56:13,202 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8041406d0a1ddc5b9409edb79457d352, store=B 2023-03-21 11:56:13,202 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:56:13,202 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8041406d0a1ddc5b9409edb79457d352, store=C 2023-03-21 11:56:13,202 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:56:13,621 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230321f9a09e0ad146419093b147728eee35f8 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230321f9a09e0ad146419093b147728eee35f8 2023-03-21 11:56:13,639 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=118, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/A/44f66262d69c48b6a05636671dd1f61c 2023-03-21 11:56:13,667 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=118 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/B/72d5721a41824a4a942803ce2aaf07c0 2023-03-21 11:56:13,692 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=118 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/C/73027b46d8c74daabaad8863499b715f 2023-03-21 11:56:13,698 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/A/44f66262d69c48b6a05636671dd1f61c as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/44f66262d69c48b6a05636671dd1f61c 2023-03-21 11:56:13,703 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/44f66262d69c48b6a05636671dd1f61c, entries=150, sequenceid=118, filesize=25.1 K 2023-03-21 11:56:13,704 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/B/72d5721a41824a4a942803ce2aaf07c0 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/72d5721a41824a4a942803ce2aaf07c0 2023-03-21 11:56:13,717 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/72d5721a41824a4a942803ce2aaf07c0, entries=150, sequenceid=118, filesize=11.6 K 2023-03-21 11:56:13,718 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/C/73027b46d8c74daabaad8863499b715f as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/73027b46d8c74daabaad8863499b715f 2023-03-21 11:56:13,731 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/73027b46d8c74daabaad8863499b715f, entries=150, sequenceid=118, filesize=11.6 K 2023-03-21 11:56:13,732 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=0 B/0 for 8041406d0a1ddc5b9409edb79457d352 in 530ms, sequenceid=118, compaction requested=true 2023-03-21 11:56:13,733 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8041406d0a1ddc5b9409edb79457d352: 2023-03-21 11:56:13,733 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. 2023-03-21 11:56:13,733 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:56:13,733 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:56:13,733 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:13,733 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:56:13,733 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:56:13,734 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:13,734 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:13,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:13,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:13,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:13,735 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:13,735 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:56:13,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:13,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:13,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:13,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:13,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:13,736 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:56:13,736 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1145002e[Count = 0] remaining members to acquire global barrier 2023-03-21 11:56:13,736 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:56:13,736 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:13,737 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:13,737 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:13,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:13,737 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:13,737 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:56:13,737 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:56:13,737 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:56:13,737 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:56:13,738 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:56:13,738 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:13,738 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:13,738 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:56:13,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:13,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:13,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:13,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:13,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:13,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:13,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:13,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:13,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:13,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:13,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:56:13,741 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:56:13,741 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:56:13,742 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:56:13,742 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:56:13,742 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:56:13,742 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:56:13,743 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:13,743 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:13,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:13,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:13,743 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:13,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:13,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:13,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:13,743 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:13,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:13,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:13,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:13,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:13,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:13,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:13,744 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:13,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:13,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:13,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:13,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:13,745 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:13,745 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:13,745 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:13,745 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:13,745 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:13,746 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:56:13,746 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:56:13,745 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:13,746 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:13,746 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:13,746 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:13,746 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:13,746 DEBUG [Thread-776] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:56:13,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:13,746 DEBUG [Thread-776] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:56:13,746 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:56:13,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:13,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:13,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:13,846 DEBUG [Thread-776] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:56:13,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:56:13,850 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:56:13,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:56:13,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:56:13,853 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:56:13,853 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:13,853 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:56:13,853 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:56:13,853 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:13,853 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:13,858 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:13,858 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:13,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:13,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:13,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:13,860 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:13,860 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:13,860 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:56:13,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:56:13,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:13,861 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:56:13,862 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:56:13,862 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:56:13,862 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:56:13,862 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:13,862 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:56:13,862 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:56:13,862 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:56:13,862 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. 2023-03-21 11:56:13,863 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. started... 2023-03-21 11:56:13,863 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8041406d0a1ddc5b9409edb79457d352: 2023-03-21 11:56:13,863 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. 2023-03-21 11:56:13,863 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:56:13,863 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:56:13,863 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:13,863 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:56:13,863 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:56:13,870 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:13,870 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:13,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:13,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:13,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:13,871 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:13,871 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:56:13,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:13,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:13,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:13,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:13,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:13,872 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:56:13,873 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ec6dc03[Count = 0] remaining members to acquire global barrier 2023-03-21 11:56:13,873 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:56:13,873 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:13,873 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:13,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:13,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:13,873 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:56:13,873 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:56:13,873 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:56:13,874 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:13,874 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:56:13,874 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:56:13,874 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:13,875 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:56:13,874 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:13,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:13,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:13,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:13,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:13,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:13,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:13,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:13,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:13,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:13,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:13,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:56:13,877 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:56:13,877 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:56:13,877 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:56:13,877 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:56:13,877 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:56:13,877 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:56:13,878 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:13,878 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:13,878 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:13,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:13,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:13,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:13,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:13,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:13,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:13,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:13,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:13,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:13,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:13,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:13,887 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:13,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:13,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:13,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:13,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:13,888 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:13,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:13,890 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:13,890 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:13,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:13,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:13,890 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:13,890 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:56:13,891 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:56:13,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:13,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:13,890 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:13,890 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:13,891 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:13,891 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:13,891 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:13,891 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:13,891 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:13,891 DEBUG [Thread-776] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:56:13,891 DEBUG [Thread-776] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:56:13,991 DEBUG [Thread-776] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:56:13,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:56:13,994 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:56:14,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:56:14,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:56:14,001 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:56:14,001 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:14,001 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:56:14,001 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:56:14,001 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:14,001 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:14,005 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:14,005 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:14,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:14,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:14,005 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:14,005 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:56:14,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:14,006 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:14,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:56:14,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:14,006 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:56:14,006 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:56:14,006 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:56:14,006 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:56:14,006 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:14,006 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:56:14,022 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:56:14,022 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. 2023-03-21 11:56:14,022 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. started... 2023-03-21 11:56:14,022 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:56:14,022 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8041406d0a1ddc5b9409edb79457d352 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-03-21 11:56:14,023 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8041406d0a1ddc5b9409edb79457d352, store=A 2023-03-21 11:56:14,023 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:56:14,023 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8041406d0a1ddc5b9409edb79457d352, store=B 2023-03-21 11:56:14,023 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:56:14,023 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8041406d0a1ddc5b9409edb79457d352, store=C 2023-03-21 11:56:14,023 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:56:14,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(9158): Flush requested on 8041406d0a1ddc5b9409edb79457d352 2023-03-21 11:56:14,039 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. as already flushing 2023-03-21 11:56:14,093 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=859, cellSize=39335, totalHeapSize=89576, min timestamp=1679399774033, max timestamp=1679399774091Num uniques 859; 2023-03-21 11:56:14,093 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=859, cellSize=39335, totalHeapSize=89576, min timestamp=1679399774033, max timestamp=1679399774091Num uniques 859; 2023-03-21 11:56:14,093 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=39335, suffix heap size=117064,new segment heap size=89576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=859 2023-03-21 11:56:14,093 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=859, cellSize=39335, totalHeapSize=89576, min timestamp=1679399774033, max timestamp=1679399774091Num uniques 859; 2023-03-21 11:56:14,093 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=39335, suffix heap size=117064,new segment heap size=89576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=859 2023-03-21 11:56:14,093 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=39335, suffix heap size=117064,new segment heap size=89576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=859 2023-03-21 11:56:14,111 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:14,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55188 deadline: 1679399834106, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:14,111 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:14,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55216 deadline: 1679399834106, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:14,112 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:14,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55180 deadline: 1679399834107, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:14,117 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:14,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55190 deadline: 1679399834108, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:14,215 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:14,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55216 deadline: 1679399834213, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:14,215 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:14,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55188 deadline: 1679399834213, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:14,216 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:14,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55180 deadline: 1679399834213, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:14,222 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:14,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55190 deadline: 1679399834219, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:14,419 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:14,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55216 deadline: 1679399834416, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:14,419 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:14,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55188 deadline: 1679399834416, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:14,419 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:14,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55180 deadline: 1679399834418, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:14,427 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:14,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55190 deadline: 1679399834424, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:14,460 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230321bc9aa8eadc1f40c1b12d993a10f5bd58 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230321bc9aa8eadc1f40c1b12d993a10f5bd58 2023-03-21 11:56:14,721 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:14,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55216 deadline: 1679399834720, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:14,721 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:14,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55188 deadline: 1679399834720, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:14,723 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:14,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55180 deadline: 1679399834722, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:14,732 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:14,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55190 deadline: 1679399834729, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:14,869 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=125, memsize=6.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/A/ee0fa6f3e3214277b04ba83e1ddebe6a 2023-03-21 11:56:15,228 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:15,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55216 deadline: 1679399835225, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:15,229 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:15,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55188 deadline: 1679399835225, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:15,229 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:15,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55180 deadline: 1679399835227, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:15,237 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:15,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55190 deadline: 1679399835235, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:15,307 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=125 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/B/f06c058f1c584497bab1483a933d8fac 2023-03-21 11:56:15,729 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=125 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/C/e5b73ee5f7b24c09954092922f0bf633 2023-03-21 11:56:15,733 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/A/ee0fa6f3e3214277b04ba83e1ddebe6a as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/ee0fa6f3e3214277b04ba83e1ddebe6a 2023-03-21 11:56:15,737 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/ee0fa6f3e3214277b04ba83e1ddebe6a, entries=150, sequenceid=125, filesize=25.1 K 2023-03-21 11:56:15,739 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/B/f06c058f1c584497bab1483a933d8fac as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/f06c058f1c584497bab1483a933d8fac 2023-03-21 11:56:15,743 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/f06c058f1c584497bab1483a933d8fac, entries=100, sequenceid=125, filesize=9.3 K 2023-03-21 11:56:15,744 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/C/e5b73ee5f7b24c09954092922f0bf633 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/e5b73ee5f7b24c09954092922f0bf633 2023-03-21 11:56:15,750 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/e5b73ee5f7b24c09954092922f0bf633, entries=100, sequenceid=125, filesize=9.3 K 2023-03-21 11:56:15,751 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.13 KB/20610, heapSize ~60.47 KB/61920, currentSize=181.14 KB/185490 for 8041406d0a1ddc5b9409edb79457d352 in 1729ms, sequenceid=125, compaction requested=true 2023-03-21 11:56:15,751 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8041406d0a1ddc5b9409edb79457d352: 2023-03-21 11:56:15,751 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. 2023-03-21 11:56:15,751 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:56:15,751 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:56:15,751 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:15,751 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:56:15,751 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:56:15,752 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:15,752 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:15,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:15,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:15,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:15,753 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:15,753 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:56:15,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:15,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:15,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:15,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:15,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:15,754 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:56:15,754 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@59bd542a[Count = 0] remaining members to acquire global barrier 2023-03-21 11:56:15,754 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:56:15,754 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:15,755 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:15,755 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:56:15,755 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:15,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:15,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:15,755 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:56:15,756 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:56:15,756 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:56:15,756 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:56:15,756 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:15,756 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:15,756 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:56:15,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:15,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:15,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:15,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:15,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:15,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:15,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:15,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:15,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:15,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:15,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:56:15,758 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:56:15,758 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:56:15,758 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:56:15,758 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:56:15,758 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:56:15,758 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:56:15,759 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:15,759 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:15,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:15,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:15,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:15,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:15,759 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:15,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:15,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:15,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:15,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:15,760 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:15,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:15,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:15,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:15,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:15,761 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:15,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:15,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:15,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:15,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:15,763 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:15,763 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:56:15,763 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:56:15,763 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:56:15,763 DEBUG [Thread-776] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:56:15,763 DEBUG [Thread-776] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:56:15,763 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:15,763 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:15,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:15,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:15,763 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:15,764 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:15,764 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:15,764 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:15,764 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:15,764 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:15,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:15,764 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:15,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:15,863 DEBUG [Thread-776] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:56:15,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:56:15,865 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:56:15,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:56:15,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:56:15,867 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:56:15,867 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:15,867 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:56:15,867 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:56:15,868 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:15,868 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:15,869 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:15,869 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:15,869 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:15,869 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:56:15,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:15,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:15,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:15,870 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:15,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:56:15,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:15,870 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:56:15,873 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:56:15,873 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:56:15,873 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:56:15,873 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:15,873 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:56:15,874 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:56:15,874 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:56:15,874 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. 2023-03-21 11:56:15,874 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. started... 2023-03-21 11:56:15,874 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8041406d0a1ddc5b9409edb79457d352 3/3 column families, dataSize=181.14 KB heapSize=458.81 KB 2023-03-21 11:56:15,874 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8041406d0a1ddc5b9409edb79457d352, store=A 2023-03-21 11:56:15,874 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-21 11:56:15,874 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8041406d0a1ddc5b9409edb79457d352, store=B 2023-03-21 11:56:15,874 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-21 11:56:15,874 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8041406d0a1ddc5b9409edb79457d352, store=C 2023-03-21 11:56:15,874 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-21 11:56:15,890 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230321eb3a1327f96444e59f5d4491c8fde291 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230321eb3a1327f96444e59f5d4491c8fde291 2023-03-21 11:56:15,894 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=155, memsize=60.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/A/7a7d86f264484f6ba58a5e72773aceeb 2023-03-21 11:56:15,906 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=60.38 KB at sequenceid=155 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/B/d91baa12a35140209bc866d667b6947f 2023-03-21 11:56:16,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(9158): Flush requested on 8041406d0a1ddc5b9409edb79457d352 2023-03-21 11:56:16,238 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. as already flushing 2023-03-21 11:56:16,246 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:16,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55180 deadline: 1679399836239, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:16,246 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:16,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55216 deadline: 1679399836241, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:16,247 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:16,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55190 deadline: 1679399836242, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:16,253 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:16,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55188 deadline: 1679399836249, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:16,341 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=60.38 KB at sequenceid=155 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/C/03e8f67e07bf471995d004b74327ff81 2023-03-21 11:56:16,345 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/A/7a7d86f264484f6ba58a5e72773aceeb as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/7a7d86f264484f6ba58a5e72773aceeb 2023-03-21 11:56:16,349 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:16,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55180 deadline: 1679399836347, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:16,355 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/7a7d86f264484f6ba58a5e72773aceeb, entries=150, sequenceid=155, filesize=25.3 K 2023-03-21 11:56:16,356 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/B/d91baa12a35140209bc866d667b6947f as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/d91baa12a35140209bc866d667b6947f 2023-03-21 11:56:16,357 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:16,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55188 deadline: 1679399836356, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:16,395 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/d91baa12a35140209bc866d667b6947f, entries=150, sequenceid=155, filesize=11.8 K 2023-03-21 11:56:16,396 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/C/03e8f67e07bf471995d004b74327ff81 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/03e8f67e07bf471995d004b74327ff81 2023-03-21 11:56:16,404 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/03e8f67e07bf471995d004b74327ff81, entries=150, sequenceid=155, filesize=11.8 K 2023-03-21 11:56:16,405 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~181.14 KB/185490, heapSize ~458.77 KB/469776, currentSize=20.13 KB/20610 for 8041406d0a1ddc5b9409edb79457d352 in 531ms, sequenceid=155, compaction requested=true 2023-03-21 11:56:16,405 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8041406d0a1ddc5b9409edb79457d352: 2023-03-21 11:56:16,405 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. 2023-03-21 11:56:16,405 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:56:16,405 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:56:16,405 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:16,405 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:56:16,405 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:56:16,406 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:16,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:16,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:16,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:16,406 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:16,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:16,407 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:16,407 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:56:16,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:16,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:16,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:16,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:16,408 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:56:16,408 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@49eddf58[Count = 0] remaining members to acquire global barrier 2023-03-21 11:56:16,408 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:56:16,408 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:16,409 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:16,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:16,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:16,409 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:56:16,409 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:56:16,409 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:56:16,410 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:16,410 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:56:16,410 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:56:16,410 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:16,410 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:16,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:16,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:16,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:16,411 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:56:16,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:16,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:16,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:16,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:16,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:16,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:16,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:16,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:56:16,414 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:56:16,414 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:56:16,414 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:56:16,414 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:56:16,414 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:56:16,414 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:56:16,415 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:16,415 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:16,415 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:16,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:16,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:16,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:16,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:16,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:16,415 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:16,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:16,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:16,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:16,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:16,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:16,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:16,416 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:16,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:16,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:16,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:16,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:16,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:16,418 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:16,418 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:16,418 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:16,419 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:56:16,419 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:56:16,418 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:16,419 DEBUG [Thread-776] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:56:16,419 DEBUG [Thread-776] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:56:16,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:16,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:16,418 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:16,419 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:16,419 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:16,419 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:16,419 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:16,419 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:16,419 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:56:16,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:16,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:16,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(9158): Flush requested on 8041406d0a1ddc5b9409edb79457d352 2023-03-21 11:56:16,446 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8041406d0a1ddc5b9409edb79457d352 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-21 11:56:16,446 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8041406d0a1ddc5b9409edb79457d352, store=A 2023-03-21 11:56:16,447 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:56:16,447 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8041406d0a1ddc5b9409edb79457d352, store=B 2023-03-21 11:56:16,447 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:56:16,447 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8041406d0a1ddc5b9409edb79457d352, store=C 2023-03-21 11:56:16,447 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:56:16,477 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230321ea4c1f5f3d26474c8846e67989897fde to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230321ea4c1f5f3d26474c8846e67989897fde 2023-03-21 11:56:16,487 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=165, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/A/9ecd9066fc3c4dc4a42e6327f7be7e91 2023-03-21 11:56:16,519 DEBUG [Thread-776] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:56:16,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:56:16,521 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:56:16,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:56:16,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:56:16,538 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:56:16,538 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:16,538 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:56:16,538 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:56:16,539 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:16,539 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:16,540 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=165 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/B/b3ff180225a542d79a46a38259b0cde5 2023-03-21 11:56:16,540 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:16,541 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:16,541 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:16,541 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:56:16,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:16,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:16,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:16,542 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:16,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:56:16,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:16,543 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:56:16,543 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:56:16,543 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:56:16,543 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:56:16,543 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:16,543 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:56:16,555 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679399776500, max timestamp=1679399776553Num uniques 859; 2023-03-21 11:56:16,555 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-21 11:56:16,555 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679399776500, max timestamp=1679399776553Num uniques 859; 2023-03-21 11:56:16,555 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-21 11:56:16,556 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679399776500, max timestamp=1679399776553Num uniques 859; 2023-03-21 11:56:16,557 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-21 11:56:16,558 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:56:16,558 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:56:16,559 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. 2023-03-21 11:56:16,559 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. started... 2023-03-21 11:56:16,559 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. as already flushing 2023-03-21 11:56:16,559 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. 2023-03-21 11:56:16,559 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:56:16,559 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:56:16,559 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:16,562 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:56:16,562 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:56:16,564 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:16,564 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:16,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:16,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:16,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:16,564 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:16,564 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:56:16,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:16,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:16,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:16,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:16,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:16,565 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:56:16,565 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7010e23b[Count = 0] remaining members to acquire global barrier 2023-03-21 11:56:16,566 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:56:16,566 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:16,571 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:16,571 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:16,571 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:56:16,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:16,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:16,571 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:56:16,572 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:56:16,572 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:56:16,573 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:16,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:16,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:16,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:16,574 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:56:16,574 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:16,574 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:56:16,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:16,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:16,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:16,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:16,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:16,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:16,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:16,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:56:16,579 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:56:16,579 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:56:16,579 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:56:16,579 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:56:16,579 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:56:16,579 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:56:16,589 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:16,589 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:16,589 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:16,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:16,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:16,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:16,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:16,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:16,589 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:16,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:16,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:16,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:16,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:16,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:16,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:16,591 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:16,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:16,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:16,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:16,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:16,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:16,595 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:16,595 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:16,596 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:16,596 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:16,596 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:16,596 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:56:16,596 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:56:16,596 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:16,596 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:16,596 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:16,596 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:16,596 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:56:16,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:16,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:16,596 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:16,596 DEBUG [Thread-776] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:56:16,596 DEBUG [Thread-776] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:56:16,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:16,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:16,599 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:16,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55188 deadline: 1679399836594, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:16,599 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:16,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55180 deadline: 1679399836595, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:16,600 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:16,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55208 deadline: 1679399836595, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:16,696 DEBUG [Thread-776] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:56:16,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:56:16,699 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:56:16,704 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:16,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55208 deadline: 1679399836701, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:16,704 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:16,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55188 deadline: 1679399836701, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:16,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:56:16,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:56:16,706 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:56:16,706 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:16,706 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:56:16,706 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:56:16,706 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:16,706 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:16,707 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:16,707 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:16,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:16,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:16,707 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:16,707 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:56:16,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:16,708 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:16,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:56:16,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:16,708 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:56:16,708 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:56:16,708 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:56:16,708 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:56:16,708 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:16,708 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:56:16,708 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:16,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55180 deadline: 1679399836701, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:16,714 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:56:16,714 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. 2023-03-21 11:56:16,714 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:56:16,714 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. started... 2023-03-21 11:56:16,714 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. as already flushing 2023-03-21 11:56:16,714 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. 2023-03-21 11:56:16,714 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:56:16,714 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:56:16,714 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:16,714 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:56:16,714 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:56:16,722 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:16,722 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:16,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:16,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:16,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:16,723 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:16,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:16,723 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:56:16,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:16,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:16,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:16,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:16,724 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:56:16,724 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@480e1efe[Count = 0] remaining members to acquire global barrier 2023-03-21 11:56:16,724 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:56:16,724 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:16,726 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:16,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:16,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:16,726 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:56:16,726 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:56:16,726 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:56:16,726 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:16,726 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:56:16,727 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:16,727 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:56:16,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:16,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:16,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:16,727 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:16,727 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:56:16,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:16,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:16,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:16,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:16,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:16,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:16,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:16,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:56:16,731 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:56:16,731 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:56:16,731 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:56:16,731 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:56:16,731 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:56:16,731 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:56:16,739 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:16,739 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:16,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:16,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:16,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:16,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:16,739 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:16,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:16,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:16,739 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:16,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:16,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:16,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:16,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:16,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:16,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:16,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:16,740 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:16,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:16,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:16,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:16,742 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:16,742 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:16,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:16,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:16,742 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:16,742 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:16,742 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:16,742 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:16,742 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:16,742 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:56:16,742 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:56:16,742 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:16,742 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:16,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:16,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:16,743 DEBUG [Thread-776] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:56:16,743 DEBUG [Thread-776] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:56:16,742 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:16,843 DEBUG [Thread-776] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:56:16,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:56:16,845 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:56:16,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:56:16,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:56:16,846 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:56:16,846 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:16,846 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:56:16,846 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:56:16,846 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:16,846 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:16,847 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:16,847 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:16,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:16,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:16,847 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:16,847 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:56:16,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:16,847 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:16,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:56:16,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:16,848 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:56:16,848 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:56:16,848 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:56:16,848 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:56:16,848 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:16,848 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:56:16,848 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:56:16,848 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:56:16,848 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. 2023-03-21 11:56:16,848 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. started... 2023-03-21 11:56:16,848 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. as already flushing 2023-03-21 11:56:16,848 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. 2023-03-21 11:56:16,848 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:56:16,848 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:56:16,848 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:16,848 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:56:16,849 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:56:16,849 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:16,849 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:16,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:16,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:16,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:16,850 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:16,850 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:56:16,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:16,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:16,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:16,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:16,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:16,851 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:56:16,851 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b211fd1[Count = 0] remaining members to acquire global barrier 2023-03-21 11:56:16,851 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:56:16,851 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:16,851 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:16,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:16,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:16,852 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:56:16,852 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:56:16,852 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:56:16,852 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:16,852 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:56:16,864 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:56:16,864 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:16,864 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:16,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:16,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:16,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:16,864 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:56:16,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:16,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:16,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:16,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:16,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:16,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:16,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:16,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:56:16,867 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:56:16,867 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:56:16,867 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:56:16,867 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:56:16,867 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:56:16,867 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:56:16,871 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:16,871 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:16,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:16,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:16,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:16,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:16,871 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:16,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:16,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:16,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:16,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:16,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:16,872 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:16,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:16,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:16,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:16,873 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:16,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:16,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:16,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:16,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:16,880 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:16,880 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:16,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:16,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:16,880 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:16,881 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:16,881 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:16,881 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:16,881 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:16,881 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:16,881 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:16,881 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:16,881 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:56:16,881 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:56:16,881 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:56:16,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:16,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:16,881 DEBUG [Thread-776] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:56:16,882 DEBUG [Thread-776] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:56:16,910 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:16,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55208 deadline: 1679399836905, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:16,911 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:16,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55188 deadline: 1679399836906, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:16,912 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:16,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55180 deadline: 1679399836910, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:16,982 DEBUG [Thread-776] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:56:16,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:56:16,984 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:56:16,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:56:16,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:56:16,986 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:56:16,986 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:16,986 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:56:16,986 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:56:16,986 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:16,986 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:16,987 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:16,987 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:16,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:16,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:16,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:16,987 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:16,987 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:56:16,988 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:16,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:56:16,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:16,988 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:56:16,988 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:56:16,988 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:56:16,988 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:56:16,988 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:16,988 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:56:16,988 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:56:16,988 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:56:16,988 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. 2023-03-21 11:56:16,989 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. started... 2023-03-21 11:56:16,989 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. as already flushing 2023-03-21 11:56:16,989 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. 2023-03-21 11:56:16,989 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:56:16,989 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:56:16,989 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:16,989 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:56:16,989 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:56:16,990 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=165 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/C/0efc776393bb4319a23f66765375ee68 2023-03-21 11:56:16,990 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:16,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:16,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:16,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:16,990 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:16,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:16,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:16,991 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:16,991 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:56:16,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:16,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:16,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:16,994 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:56:16,994 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:56:16,994 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c1108f2[Count = 0] remaining members to acquire global barrier 2023-03-21 11:56:16,994 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:16,999 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/A/9ecd9066fc3c4dc4a42e6327f7be7e91 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/9ecd9066fc3c4dc4a42e6327f7be7e91 2023-03-21 11:56:17,003 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/9ecd9066fc3c4dc4a42e6327f7be7e91, entries=200, sequenceid=165, filesize=32.1 K 2023-03-21 11:56:17,004 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:17,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:17,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:17,004 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:56:17,004 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:17,004 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:56:17,004 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:56:17,004 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:56:17,005 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/B/b3ff180225a542d79a46a38259b0cde5 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/b3ff180225a542d79a46a38259b0cde5 2023-03-21 11:56:17,006 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:17,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:17,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:17,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:17,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:17,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:17,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:17,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:17,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:17,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:17,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:17,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:56:17,008 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:56:17,008 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:56:17,008 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:56:17,008 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:56:17,008 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:56:17,008 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:56:17,009 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:17,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:17,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:17,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:17,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:17,009 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:17,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:17,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:17,010 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:56:17,010 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:17,010 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:17,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:17,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:17,010 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:56:17,010 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:17,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:17,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:17,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:17,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:17,011 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:17,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:17,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:17,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:17,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:17,015 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:17,015 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:17,015 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:17,015 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:17,015 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:56:17,015 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:56:17,015 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/b3ff180225a542d79a46a38259b0cde5, entries=150, sequenceid=165, filesize=11.8 K 2023-03-21 11:56:17,015 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:17,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:17,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:17,015 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:17,015 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:17,015 DEBUG [Thread-776] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:56:17,015 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:17,015 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:17,015 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:17,015 DEBUG [Thread-776] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:56:17,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:17,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:17,016 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/C/0efc776393bb4319a23f66765375ee68 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/0efc776393bb4319a23f66765375ee68 2023-03-21 11:56:17,020 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/0efc776393bb4319a23f66765375ee68, entries=150, sequenceid=165, filesize=11.8 K 2023-03-21 11:56:17,021 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=140.89 KB/144270 for 8041406d0a1ddc5b9409edb79457d352 in 575ms, sequenceid=165, compaction requested=true 2023-03-21 11:56:17,021 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8041406d0a1ddc5b9409edb79457d352: 2023-03-21 11:56:17,021 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-21 11:56:17,021 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 9 store files, 0 compacting, 9 eligible, 16 blocking 2023-03-21 11:56:17,021 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 9 store files, 0 compacting, 9 eligible, 16 blocking 2023-03-21 11:56:17,021 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-21 11:56:17,021 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-21 11:56:17,030 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 9 files of size 232052 starting at candidate #0 after considering 28 permutations with 28 in ratio 2023-03-21 11:56:17,030 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1912): 8041406d0a1ddc5b9409edb79457d352/A is initiating minor compaction (all files) 2023-03-21 11:56:17,030 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8041406d0a1ddc5b9409edb79457d352/A in TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. 2023-03-21 11:56:17,030 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/b1457dfd3fc0408281a28e09e36d2169, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/7c1af716a9f848afa158ef97aea37252, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/27e21010df214d5c8ea61b10d07a0e7c, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/47c811f81290442490e545740b1bd5a1, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/83e40448e5a6453bb520cb8813d3fb16, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/44f66262d69c48b6a05636671dd1f61c, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/ee0fa6f3e3214277b04ba83e1ddebe6a, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/7a7d86f264484f6ba58a5e72773aceeb, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/9ecd9066fc3c4dc4a42e6327f7be7e91] into tmpdir=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp, totalSize=226.6 K 2023-03-21 11:56:17,031 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 9 files of size 102586 starting at candidate #0 after considering 28 permutations with 28 in ratio 2023-03-21 11:56:17,031 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting b1457dfd3fc0408281a28e09e36d2169, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1679399767401 2023-03-21 11:56:17,031 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1912): 8041406d0a1ddc5b9409edb79457d352/B is initiating minor compaction (all files) 2023-03-21 11:56:17,031 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8041406d0a1ddc5b9409edb79457d352/B in TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. 2023-03-21 11:56:17,031 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/40ba77cc4ae04c7495923c06f5df12d9, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/6d4b7ace0bd54798b215a54de28b75cb, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/05d3c84401744612ba8ac5a5563e06cd, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/a47ca18fe7904b6a9d572f9a8bacdaed, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/eb3995f2026d4b9b8790df5a88aa495f, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/72d5721a41824a4a942803ce2aaf07c0, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/f06c058f1c584497bab1483a933d8fac, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/d91baa12a35140209bc866d667b6947f, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/b3ff180225a542d79a46a38259b0cde5] into tmpdir=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp, totalSize=100.2 K 2023-03-21 11:56:17,031 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 7c1af716a9f848afa158ef97aea37252, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=53, earliestPutTs=1679399767474 2023-03-21 11:56:17,032 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 40ba77cc4ae04c7495923c06f5df12d9, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1679399767401 2023-03-21 11:56:17,032 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 27e21010df214d5c8ea61b10d07a0e7c, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=72, earliestPutTs=1679399768202 2023-03-21 11:56:17,032 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 6d4b7ace0bd54798b215a54de28b75cb, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=53, earliestPutTs=1679399767474 2023-03-21 11:56:17,032 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 47c811f81290442490e545740b1bd5a1, keycount=100, bloomtype=ROW, size=18.4 K, encoding=NONE, compression=NONE, seqNum=85, earliestPutTs=1679399770342 2023-03-21 11:56:17,033 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 05d3c84401744612ba8ac5a5563e06cd, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=72, earliestPutTs=1679399768202 2023-03-21 11:56:17,033 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 83e40448e5a6453bb520cb8813d3fb16, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=105, earliestPutTs=1679399771514 2023-03-21 11:56:17,033 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting a47ca18fe7904b6a9d572f9a8bacdaed, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=85, earliestPutTs=1679399770342 2023-03-21 11:56:17,033 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 44f66262d69c48b6a05636671dd1f61c, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=118, earliestPutTs=1679399771842 2023-03-21 11:56:17,033 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting eb3995f2026d4b9b8790df5a88aa495f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=105, earliestPutTs=1679399771514 2023-03-21 11:56:17,034 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting ee0fa6f3e3214277b04ba83e1ddebe6a, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=125, earliestPutTs=1679399773983 2023-03-21 11:56:17,034 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 72d5721a41824a4a942803ce2aaf07c0, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=118, earliestPutTs=1679399771842 2023-03-21 11:56:17,035 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting f06c058f1c584497bab1483a933d8fac, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=125, earliestPutTs=1679399773983 2023-03-21 11:56:17,035 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 7a7d86f264484f6ba58a5e72773aceeb, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=155, earliestPutTs=1679399774104 2023-03-21 11:56:17,035 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting d91baa12a35140209bc866d667b6947f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=155, earliestPutTs=1679399774104 2023-03-21 11:56:17,036 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 9ecd9066fc3c4dc4a42e6327f7be7e91, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=165, earliestPutTs=1679399776245 2023-03-21 11:56:17,037 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting b3ff180225a542d79a46a38259b0cde5, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=165, earliestPutTs=1679399776245 2023-03-21 11:56:17,077 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] throttle.PressureAwareThroughputController(145): 8041406d0a1ddc5b9409edb79457d352#B#compaction#118 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-21 11:56:17,100 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8041406d0a1ddc5b9409edb79457d352#A#compaction#117 average throughput is 0.62 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-21 11:56:17,115 DEBUG [Thread-776] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:56:17,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:56:17,119 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:56:17,124 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/B/be0be94d0dba4e219c17a9369250d1f2 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/be0be94d0dba4e219c17a9369250d1f2 2023-03-21 11:56:17,146 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1652): Completed compaction of 9 (all) file(s) in 8041406d0a1ddc5b9409edb79457d352/B of 8041406d0a1ddc5b9409edb79457d352 into be0be94d0dba4e219c17a9369250d1f2(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-21 11:56:17,147 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8041406d0a1ddc5b9409edb79457d352: 2023-03-21 11:56:17,147 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352., storeName=8041406d0a1ddc5b9409edb79457d352/B, priority=7, startTime=1679399777021; duration=0sec 2023-03-21 11:56:17,147 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-21 11:56:17,147 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 9 store files, 0 compacting, 9 eligible, 16 blocking 2023-03-21 11:56:17,155 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 9 files of size 102586 starting at candidate #0 after considering 28 permutations with 28 in ratio 2023-03-21 11:56:17,155 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1912): 8041406d0a1ddc5b9409edb79457d352/C is initiating minor compaction (all files) 2023-03-21 11:56:17,156 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8041406d0a1ddc5b9409edb79457d352/C in TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. 2023-03-21 11:56:17,156 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/f3cdb56e0bdb4a9bbb57038bfd3f1f8b, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/c144e9fdcfbc498bb26bce99c8df5d7c, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/93e36abfb16c4fd88667a1278a8b2e5a, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/77dc6fa1beac42d4ae2ca72b468e18dc, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/6264dd4f5b884048bdf3868360a8bd7f, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/73027b46d8c74daabaad8863499b715f, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/e5b73ee5f7b24c09954092922f0bf633, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/03e8f67e07bf471995d004b74327ff81, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/0efc776393bb4319a23f66765375ee68] into tmpdir=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp, totalSize=100.2 K 2023-03-21 11:56:17,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:56:17,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:56:17,156 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:56:17,156 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:17,157 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:56:17,157 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:56:17,157 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:17,157 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:17,158 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:17,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:17,159 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:17,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:17,159 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:17,159 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:56:17,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:17,160 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:17,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:56:17,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:17,160 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:56:17,163 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting f3cdb56e0bdb4a9bbb57038bfd3f1f8b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1679399767401 2023-03-21 11:56:17,164 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:56:17,164 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:56:17,164 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:56:17,164 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:17,164 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:56:17,165 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting c144e9fdcfbc498bb26bce99c8df5d7c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=53, earliestPutTs=1679399767474 2023-03-21 11:56:17,166 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 93e36abfb16c4fd88667a1278a8b2e5a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=72, earliestPutTs=1679399768202 2023-03-21 11:56:17,167 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 77dc6fa1beac42d4ae2ca72b468e18dc, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=85, earliestPutTs=1679399770342 2023-03-21 11:56:17,168 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 6264dd4f5b884048bdf3868360a8bd7f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=105, earliestPutTs=1679399771514 2023-03-21 11:56:17,168 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 73027b46d8c74daabaad8863499b715f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=118, earliestPutTs=1679399771842 2023-03-21 11:56:17,169 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting e5b73ee5f7b24c09954092922f0bf633, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=125, earliestPutTs=1679399773983 2023-03-21 11:56:17,172 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:56:17,172 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. 2023-03-21 11:56:17,173 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 03e8f67e07bf471995d004b74327ff81, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=155, earliestPutTs=1679399774104 2023-03-21 11:56:17,172 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:56:17,176 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. started... 2023-03-21 11:56:17,176 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8041406d0a1ddc5b9409edb79457d352 3/3 column families, dataSize=140.89 KB heapSize=395.53 KB 2023-03-21 11:56:17,176 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8041406d0a1ddc5b9409edb79457d352, store=A 2023-03-21 11:56:17,178 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-21 11:56:17,178 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8041406d0a1ddc5b9409edb79457d352, store=B 2023-03-21 11:56:17,178 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-21 11:56:17,178 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8041406d0a1ddc5b9409edb79457d352, store=C 2023-03-21 11:56:17,178 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 0efc776393bb4319a23f66765375ee68, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=165, earliestPutTs=1679399776245 2023-03-21 11:56:17,179 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-21 11:56:17,211 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] throttle.PressureAwareThroughputController(145): 8041406d0a1ddc5b9409edb79457d352#C#compaction#120 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-21 11:56:17,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(9158): Flush requested on 8041406d0a1ddc5b9409edb79457d352 2023-03-21 11:56:17,215 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. as already flushing 2023-03-21 11:56:17,222 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230321a81b6a64603a4277a08b011999e6caec to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230321a81b6a64603a4277a08b011999e6caec 2023-03-21 11:56:17,251 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:17,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55208 deadline: 1679399837243, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:17,251 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:17,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55180 deadline: 1679399837246, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:17,255 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:17,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55188 deadline: 1679399837251, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:17,353 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:17,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55208 deadline: 1679399837352, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:17,353 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:17,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55180 deadline: 1679399837352, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:17,363 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:17,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55188 deadline: 1679399837357, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:17,556 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:17,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55208 deadline: 1679399837555, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:17,560 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:17,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55180 deadline: 1679399837556, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:17,569 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:17,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55188 deadline: 1679399837565, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:17,629 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=202, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/A/ddd9bddd917c4463a3d0cec5fe0b0f26 2023-03-21 11:56:17,633 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/C/f86db8c925ef4b43a1ab01df1e57e2df as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/f86db8c925ef4b43a1ab01df1e57e2df 2023-03-21 11:56:17,639 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1652): Completed compaction of 9 (all) file(s) in 8041406d0a1ddc5b9409edb79457d352/C of 8041406d0a1ddc5b9409edb79457d352 into f86db8c925ef4b43a1ab01df1e57e2df(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-21 11:56:17,639 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8041406d0a1ddc5b9409edb79457d352: 2023-03-21 11:56:17,639 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352., storeName=8041406d0a1ddc5b9409edb79457d352/C, priority=7, startTime=1679399777021; duration=0sec 2023-03-21 11:56:17,639 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-21 11:56:17,859 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:17,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55208 deadline: 1679399837858, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:17,862 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:17,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55180 deadline: 1679399837861, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:17,873 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:17,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55188 deadline: 1679399837872, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:17,996 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/A/273a452d436748db9b85a0a70c1d5ad1 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/273a452d436748db9b85a0a70c1d5ad1 2023-03-21 11:56:18,003 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 9 (all) file(s) in 8041406d0a1ddc5b9409edb79457d352/A of 8041406d0a1ddc5b9409edb79457d352 into 273a452d436748db9b85a0a70c1d5ad1(size=25.6 K), total size for store is 25.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-21 11:56:18,004 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8041406d0a1ddc5b9409edb79457d352: 2023-03-21 11:56:18,004 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352., storeName=8041406d0a1ddc5b9409edb79457d352/A, priority=7, startTime=1679399777021; duration=0sec 2023-03-21 11:56:18,004 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-21 11:56:18,041 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=202 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/B/dc4187adff83447e9f945ac9ec074ce7 2023-03-21 11:56:18,062 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=202 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/C/ca9222ca5e6e4f3785f88820721dc327 2023-03-21 11:56:18,073 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/A/ddd9bddd917c4463a3d0cec5fe0b0f26 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/ddd9bddd917c4463a3d0cec5fe0b0f26 2023-03-21 11:56:18,077 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/ddd9bddd917c4463a3d0cec5fe0b0f26, entries=150, sequenceid=202, filesize=25.3 K 2023-03-21 11:56:18,078 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/B/dc4187adff83447e9f945ac9ec074ce7 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/dc4187adff83447e9f945ac9ec074ce7 2023-03-21 11:56:18,082 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/dc4187adff83447e9f945ac9ec074ce7, entries=150, sequenceid=202, filesize=11.8 K 2023-03-21 11:56:18,083 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/C/ca9222ca5e6e4f3785f88820721dc327 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/ca9222ca5e6e4f3785f88820721dc327 2023-03-21 11:56:18,088 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/ca9222ca5e6e4f3785f88820721dc327, entries=150, sequenceid=202, filesize=11.8 K 2023-03-21 11:56:18,088 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~395.48 KB/404976, currentSize=40.25 KB/41220 for 8041406d0a1ddc5b9409edb79457d352 in 912ms, sequenceid=202, compaction requested=false 2023-03-21 11:56:18,089 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8041406d0a1ddc5b9409edb79457d352: 2023-03-21 11:56:18,089 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. 2023-03-21 11:56:18,089 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:56:18,089 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:56:18,089 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:18,089 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:56:18,089 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:56:18,092 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:18,094 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:18,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:18,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:18,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:18,094 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:18,096 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:56:18,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:18,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:18,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:18,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:18,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:18,098 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:56:18,098 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b672871[Count = 0] remaining members to acquire global barrier 2023-03-21 11:56:18,098 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:56:18,098 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:18,105 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:18,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:18,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:18,105 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:18,106 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:56:18,105 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:56:18,106 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:56:18,106 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:56:18,108 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:56:18,108 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:18,108 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:18,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:18,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:18,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:18,108 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:56:18,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:18,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:18,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:18,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:18,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:18,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:18,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:18,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:56:18,112 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:56:18,112 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:56:18,112 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:56:18,112 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:56:18,112 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:56:18,112 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:56:18,121 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:18,121 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:18,121 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:18,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:18,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:18,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:18,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:18,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:18,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:18,121 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:18,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:18,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:18,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:18,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:18,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:18,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:18,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:18,123 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:18,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:18,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:18,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:18,125 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:18,125 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:18,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:18,125 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:18,125 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:18,125 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:56:18,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:18,125 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:18,125 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:18,125 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:18,125 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:18,125 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:18,125 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:18,125 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:56:18,125 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:56:18,125 DEBUG [Thread-776] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:56:18,125 DEBUG [Thread-776] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:56:18,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:18,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:18,226 DEBUG [Thread-776] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:56:18,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:56:18,228 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:56:18,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:56:18,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:56:18,229 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:56:18,229 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:18,229 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:56:18,229 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:56:18,230 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:18,230 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:18,231 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:18,234 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:18,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:18,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:18,235 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:18,235 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:56:18,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:18,235 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:18,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:56:18,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:18,236 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:56:18,236 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:56:18,236 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:56:18,236 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:56:18,236 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:18,236 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:56:18,238 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:56:18,238 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:56:18,238 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. 2023-03-21 11:56:18,238 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. started... 2023-03-21 11:56:18,238 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8041406d0a1ddc5b9409edb79457d352 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-03-21 11:56:18,238 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8041406d0a1ddc5b9409edb79457d352, store=A 2023-03-21 11:56:18,239 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:56:18,239 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8041406d0a1ddc5b9409edb79457d352, store=B 2023-03-21 11:56:18,239 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:56:18,239 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8041406d0a1ddc5b9409edb79457d352, store=C 2023-03-21 11:56:18,239 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:56:18,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(9158): Flush requested on 8041406d0a1ddc5b9409edb79457d352 2023-03-21 11:56:18,263 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. as already flushing 2023-03-21 11:56:18,285 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303211063d2f19581495f9e442d2ef2fa8d76 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303211063d2f19581495f9e442d2ef2fa8d76 2023-03-21 11:56:18,299 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=213, memsize=13.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/A/73626e4eaa22461cb38d637952b66fb0 2023-03-21 11:56:18,317 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679399778300, max timestamp=1679399778315Num uniques 859; 2023-03-21 11:56:18,317 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-21 11:56:18,317 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679399778300, max timestamp=1679399778315Num uniques 859; 2023-03-21 11:56:18,318 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-21 11:56:18,318 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679399778300, max timestamp=1679399778315Num uniques 859; 2023-03-21 11:56:18,318 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-21 11:56:18,327 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=213 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/B/b9c74a7141ed44b990ae5731c4af571f 2023-03-21 11:56:18,339 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=213 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/C/27c6a44ae8264084af2ab9acb81dc3e7 2023-03-21 11:56:18,343 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/A/73626e4eaa22461cb38d637952b66fb0 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/73626e4eaa22461cb38d637952b66fb0 2023-03-21 11:56:18,350 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/73626e4eaa22461cb38d637952b66fb0, entries=100, sequenceid=213, filesize=18.5 K 2023-03-21 11:56:18,351 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/B/b9c74a7141ed44b990ae5731c4af571f as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/b9c74a7141ed44b990ae5731c4af571f 2023-03-21 11:56:18,355 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/b9c74a7141ed44b990ae5731c4af571f, entries=100, sequenceid=213, filesize=9.4 K 2023-03-21 11:56:18,356 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/C/27c6a44ae8264084af2ab9acb81dc3e7 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/27c6a44ae8264084af2ab9acb81dc3e7 2023-03-21 11:56:18,360 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1679399778347, max timestamp=1679399778359Num uniques 1068; 2023-03-21 11:56:18,360 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=9978, suffix heap size=139040,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=218 2023-03-21 11:56:18,361 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1679399778347, max timestamp=1679399778359Num uniques 1068; 2023-03-21 11:56:18,361 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1679399778347, max timestamp=1679399778359Num uniques 1068; 2023-03-21 11:56:18,361 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=9978, suffix heap size=139040,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=218 2023-03-21 11:56:18,361 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-03-21 11:56:18,365 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/27c6a44ae8264084af2ab9acb81dc3e7, entries=100, sequenceid=213, filesize=9.4 K 2023-03-21 11:56:18,366 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=51.44 KB/52670 for 8041406d0a1ddc5b9409edb79457d352 in 128ms, sequenceid=213, compaction requested=true 2023-03-21 11:56:18,366 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8041406d0a1ddc5b9409edb79457d352: 2023-03-21 11:56:18,366 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. 2023-03-21 11:56:18,366 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:56:18,366 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:56:18,366 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:18,366 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:56:18,366 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:56:18,367 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:18,368 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:18,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:18,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:18,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:18,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:18,368 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:18,368 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:56:18,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:18,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:18,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:18,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:18,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(9158): Flush requested on 8041406d0a1ddc5b9409edb79457d352 2023-03-21 11:56:18,370 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8041406d0a1ddc5b9409edb79457d352 3/3 column families, dataSize=58.14 KB heapSize=155.23 KB 2023-03-21 11:56:18,370 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:56:18,370 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@13262a0d[Count = 0] remaining members to acquire global barrier 2023-03-21 11:56:18,370 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:56:18,370 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:18,371 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8041406d0a1ddc5b9409edb79457d352, store=A 2023-03-21 11:56:18,371 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-21 11:56:18,371 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:18,371 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8041406d0a1ddc5b9409edb79457d352, store=B 2023-03-21 11:56:18,371 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:56:18,371 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-21 11:56:18,371 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8041406d0a1ddc5b9409edb79457d352, store=C 2023-03-21 11:56:18,371 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-21 11:56:18,371 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:18,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:18,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:18,372 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:56:18,372 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:56:18,372 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:56:18,373 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:56:18,373 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:18,373 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:18,373 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:56:18,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:18,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:18,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:18,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:18,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:18,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:18,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:18,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:18,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:18,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:18,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:56:18,376 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:56:18,376 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:56:18,376 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:56:18,376 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:56:18,376 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:56:18,376 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:56:18,377 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:18,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:18,377 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:18,377 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:18,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:18,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:18,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:18,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:18,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:18,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:18,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:18,378 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:18,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:18,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:18,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:18,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:18,380 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:18,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:18,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:18,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:18,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:18,389 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:18,389 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:18,389 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:18,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:18,389 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:56:18,389 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:56:18,389 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:18,389 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:18,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:18,389 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:56:18,389 DEBUG [Thread-776] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:56:18,389 DEBUG [Thread-776] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:56:18,389 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:18,389 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:18,389 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:18,389 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:18,389 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:18,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:18,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:18,399 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:18,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55216 deadline: 1679399838392, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:18,400 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:18,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55188 deadline: 1679399838392, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:18,405 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:18,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55208 deadline: 1679399838397, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:18,406 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:18,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55190 deadline: 1679399838398, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:18,407 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:18,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55180 deadline: 1679399838400, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:18,414 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303211c84657477304428bb53560bfd92d6a0 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303211c84657477304428bb53560bfd92d6a0 2023-03-21 11:56:18,489 DEBUG [Thread-776] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:56:18,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:56:18,491 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:56:18,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:56:18,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:56:18,498 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:56:18,498 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:18,498 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:56:18,498 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:56:18,499 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:18,499 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:18,499 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:18,499 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:18,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:18,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:18,500 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:18,500 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:56:18,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:18,500 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:18,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:56:18,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:18,501 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:56:18,501 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:56:18,501 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:56:18,501 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:56:18,501 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:18,501 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:56:18,501 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:56:18,501 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:56:18,502 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. 2023-03-21 11:56:18,502 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. started... 2023-03-21 11:56:18,502 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. as already flushing 2023-03-21 11:56:18,502 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. 2023-03-21 11:56:18,502 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:56:18,502 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:56:18,502 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:18,502 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:56:18,502 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:56:18,503 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:18,503 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:18,504 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:56:18,504 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:18,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55216 deadline: 1679399838501, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:18,504 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:18,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55188 deadline: 1679399838502, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:18,504 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:18,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:18,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:18,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:18,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:18,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:18,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:18,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:18,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:18,508 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:56:18,508 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@600a602f[Count = 0] remaining members to acquire global barrier 2023-03-21 11:56:18,508 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:56:18,508 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:18,508 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:18,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55208 deadline: 1679399838507, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:18,508 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:18,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55190 deadline: 1679399838507, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:18,509 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:18,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55180 deadline: 1679399838508, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:18,509 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:18,509 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:18,509 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:56:18,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:18,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:18,509 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:56:18,509 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:56:18,510 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:56:18,510 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:56:18,510 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:18,510 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:56:18,511 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:18,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:18,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:18,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:18,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:18,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:18,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:18,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:18,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:18,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:18,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:18,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:56:18,513 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:56:18,513 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:56:18,514 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:56:18,514 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:56:18,514 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:56:18,514 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:56:18,515 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:18,515 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:18,515 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:18,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:18,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:18,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:18,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:18,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:18,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:18,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:18,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:18,518 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:18,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:18,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:18,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:18,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:18,519 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:18,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:18,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:18,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:18,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:18,529 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:18,529 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:18,529 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:56:18,529 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:18,529 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:18,529 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:18,529 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:18,529 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:18,529 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:18,529 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:18,529 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:18,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:18,529 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:56:18,529 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:56:18,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:18,530 DEBUG [Thread-776] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:56:18,530 DEBUG [Thread-776] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:56:18,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:18,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:18,630 DEBUG [Thread-776] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:56:18,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:56:18,633 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:56:18,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:56:18,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:56:18,635 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:56:18,635 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:18,635 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:56:18,635 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:56:18,635 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:18,635 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:18,636 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:18,636 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:18,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:18,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:18,636 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:18,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:18,636 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:56:18,637 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:18,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:56:18,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:18,637 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:56:18,637 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:56:18,637 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:56:18,637 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:56:18,637 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:18,637 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:56:18,637 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:56:18,638 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:56:18,638 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. 2023-03-21 11:56:18,638 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. started... 2023-03-21 11:56:18,638 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. as already flushing 2023-03-21 11:56:18,638 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. 2023-03-21 11:56:18,638 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:56:18,638 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:56:18,638 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:18,638 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:56:18,638 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:56:18,639 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:18,640 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:18,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:18,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:18,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:18,640 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:18,640 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:56:18,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:18,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:18,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:18,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:18,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:18,641 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:56:18,641 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:56:18,641 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3408b244[Count = 0] remaining members to acquire global barrier 2023-03-21 11:56:18,641 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:18,642 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:18,642 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:56:18,642 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:18,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:18,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:18,642 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:56:18,642 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:56:18,642 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:56:18,643 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:18,643 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:56:18,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:18,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:18,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:18,643 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:18,643 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:56:18,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:18,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:18,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:18,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:18,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:18,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:18,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:18,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:56:18,645 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:56:18,645 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:56:18,645 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:56:18,645 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:56:18,645 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:56:18,645 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:56:18,646 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:18,646 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:18,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:18,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:18,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:18,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:18,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:18,646 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:18,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:18,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:18,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:18,646 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:18,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:18,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:18,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:18,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:18,648 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:18,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:18,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:18,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:18,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:18,650 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:18,650 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:18,650 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:18,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:18,650 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:18,650 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:56:18,650 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:18,650 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:56:18,651 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:56:18,650 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:18,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:18,651 DEBUG [Thread-776] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:56:18,651 DEBUG [Thread-776] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:56:18,650 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:18,651 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:18,651 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:18,651 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:18,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:18,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:18,707 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:18,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55188 deadline: 1679399838706, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:18,709 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:18,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55216 deadline: 1679399838708, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:18,713 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:18,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55190 deadline: 1679399838710, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:18,713 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:18,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55180 deadline: 1679399838710, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:18,713 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:18,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55208 deadline: 1679399838710, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:18,751 DEBUG [Thread-776] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:56:18,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:56:18,754 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:56:18,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:56:18,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:56:18,755 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:56:18,755 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:18,755 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:56:18,755 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:56:18,756 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:18,756 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:18,757 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:18,757 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:18,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:18,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:18,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:18,759 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:18,759 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:56:18,760 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:18,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:56:18,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:18,766 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:56:18,774 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:56:18,774 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:56:18,774 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:56:18,774 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:18,774 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:56:18,782 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:56:18,782 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:56:18,782 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. 2023-03-21 11:56:18,782 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. started... 2023-03-21 11:56:18,782 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. as already flushing 2023-03-21 11:56:18,782 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. 2023-03-21 11:56:18,782 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:56:18,782 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:56:18,782 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:18,782 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:56:18,782 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:56:18,784 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:18,784 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:18,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:18,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:18,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:18,784 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:18,784 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:56:18,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:18,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:18,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:18,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:18,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:18,786 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:56:18,786 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@26be2014[Count = 0] remaining members to acquire global barrier 2023-03-21 11:56:18,786 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:56:18,786 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:18,787 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:18,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:18,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:18,787 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:56:18,787 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:56:18,787 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:56:18,787 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:18,787 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:56:18,788 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:18,788 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:56:18,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:18,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:18,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:18,788 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:18,788 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:56:18,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:18,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:18,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:18,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:18,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:18,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:18,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:18,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:56:18,790 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:56:18,790 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:56:18,790 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:56:18,790 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:56:18,790 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:56:18,790 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:56:18,792 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:18,792 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:18,792 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:18,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:18,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:18,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:18,793 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:18,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:18,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:18,794 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:18,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:18,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:18,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:18,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:18,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:18,796 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:18,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:18,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:18,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:18,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:18,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:18,804 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:18,804 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:18,804 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:18,804 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:18,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:18,804 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:56:18,804 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:56:18,804 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:18,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:18,804 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:18,804 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:18,804 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:56:18,804 DEBUG [Thread-776] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:56:18,804 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:18,804 DEBUG [Thread-776] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:56:18,804 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:18,804 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:18,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:18,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:18,836 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=256, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/A/bd20561c9bba4179b307acf694437b6c 2023-03-21 11:56:18,904 DEBUG [Thread-776] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:56:18,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:56:18,906 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:56:18,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:56:18,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:56:18,907 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:56:18,907 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:18,907 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:56:18,907 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:56:18,908 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:18,908 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:18,908 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:18,908 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:18,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:18,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:18,909 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:18,909 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:56:18,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:18,909 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:18,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:56:18,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:18,909 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:56:18,909 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:56:18,909 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:56:18,909 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:56:18,909 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:18,909 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:56:18,910 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:56:18,910 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:56:18,910 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. 2023-03-21 11:56:18,910 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. started... 2023-03-21 11:56:18,910 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. as already flushing 2023-03-21 11:56:18,910 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. 2023-03-21 11:56:18,910 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:56:18,910 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:56:18,910 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:18,910 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:56:18,910 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:56:18,911 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:18,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:18,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:18,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:18,911 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:18,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:18,911 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:18,911 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:56:18,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:18,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:18,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:18,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:18,912 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:56:18,912 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4ea108c[Count = 0] remaining members to acquire global barrier 2023-03-21 11:56:18,912 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:56:18,913 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:18,913 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:18,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:18,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:18,913 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:56:18,913 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:56:18,913 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:56:18,913 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:18,913 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:56:18,914 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:18,914 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:56:18,914 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:18,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:18,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:18,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:18,914 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:56:18,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:18,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:18,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:18,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:18,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:18,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:18,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:18,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:56:18,917 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:56:18,917 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:56:18,917 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:56:18,917 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:56:18,917 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:56:18,917 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:56:18,933 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:18,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:18,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:18,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:18,934 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:18,934 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:18,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:18,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:18,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:18,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:18,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:18,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:18,936 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:18,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:18,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:18,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:18,937 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:18,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:18,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:18,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:18,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:18,938 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:18,938 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:56:18,938 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:18,938 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:56:18,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:18,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:18,938 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:18,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:18,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:18,939 DEBUG [Thread-776] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:56:18,939 DEBUG [Thread-776] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:56:18,939 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:56:18,942 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:18,942 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:18,942 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:18,942 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:18,942 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:18,942 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:18,942 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:19,013 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:19,013 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:19,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55188 deadline: 1679399839010, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:19,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55216 deadline: 1679399839011, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:19,014 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:19,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55208 deadline: 1679399839014, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:19,018 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:19,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55180 deadline: 1679399839016, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:19,019 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:19,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55190 deadline: 1679399839016, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:19,039 DEBUG [Thread-776] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:56:19,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:56:19,041 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:56:19,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:56:19,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:56:19,042 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:56:19,042 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:19,042 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:56:19,042 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:56:19,042 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:19,042 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:19,043 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:19,043 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:19,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:19,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:19,043 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:19,043 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:56:19,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:19,044 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:19,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:56:19,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:19,044 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:56:19,045 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:56:19,045 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:56:19,045 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:19,045 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:56:19,045 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:56:19,045 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:56:19,045 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. 2023-03-21 11:56:19,045 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. started... 2023-03-21 11:56:19,045 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. as already flushing 2023-03-21 11:56:19,045 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. 2023-03-21 11:56:19,045 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:56:19,045 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:56:19,045 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:19,045 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:56:19,045 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:56:19,046 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:19,046 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:19,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:19,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:19,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:19,046 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:19,046 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:56:19,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:19,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:19,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:19,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:19,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:19,049 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:56:19,049 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@37520326[Count = 0] remaining members to acquire global barrier 2023-03-21 11:56:19,049 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:56:19,049 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:19,049 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:19,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:19,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:19,049 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:56:19,049 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:56:19,049 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:56:19,050 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:19,050 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:56:19,050 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:19,050 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:56:19,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:19,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:19,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:19,050 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:19,051 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:56:19,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:19,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:19,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:19,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:19,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:19,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:19,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:19,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:56:19,052 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:56:19,052 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:56:19,052 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:56:19,052 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:56:19,052 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:56:19,052 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:56:19,053 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:19,053 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:19,053 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:19,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:19,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:19,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:19,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:19,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:19,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:19,054 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:19,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:19,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:19,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:19,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:19,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:19,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:19,055 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:19,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:19,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:19,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:19,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:19,057 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:19,057 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:56:19,057 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:56:19,058 DEBUG [Thread-776] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:56:19,058 DEBUG [Thread-776] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:56:19,058 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:19,058 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:19,058 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:19,058 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:19,058 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:19,058 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:19,058 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:19,058 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:56:19,058 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:19,058 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:19,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:19,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:19,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:19,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:19,158 DEBUG [Thread-776] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:56:19,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:56:19,160 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:56:19,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:56:19,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:56:19,162 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:56:19,162 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:19,162 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:56:19,162 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:56:19,163 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:19,163 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:19,164 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:19,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:19,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:19,164 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:19,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:19,164 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:19,164 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:56:19,166 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:19,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:56:19,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:19,166 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:56:19,173 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:56:19,174 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:56:19,174 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:56:19,174 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:19,174 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:56:19,174 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:56:19,174 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:56:19,174 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. 2023-03-21 11:56:19,174 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. started... 2023-03-21 11:56:19,174 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. as already flushing 2023-03-21 11:56:19,174 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. 2023-03-21 11:56:19,174 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:56:19,174 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:56:19,174 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:19,175 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:56:19,175 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:56:19,183 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:19,183 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:19,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:19,183 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:19,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:19,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:19,184 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:56:19,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:19,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:19,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:19,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:19,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:19,185 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:56:19,185 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@46391d1a[Count = 0] remaining members to acquire global barrier 2023-03-21 11:56:19,185 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:56:19,185 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:19,186 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:19,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:19,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:19,186 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:56:19,186 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:56:19,186 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:56:19,187 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:19,187 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:56:19,187 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:19,187 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:56:19,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:19,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:19,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:19,187 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:19,187 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:56:19,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:19,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:19,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:19,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:19,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:19,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:19,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:19,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:56:19,191 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:56:19,191 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:56:19,191 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:56:19,191 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:56:19,191 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:56:19,191 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:56:19,192 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:19,192 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:19,192 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:19,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:19,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:19,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:19,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:19,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:19,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:19,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:19,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:19,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:19,194 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:19,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:19,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:19,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:19,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:19,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:19,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:19,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:19,198 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:19,206 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:19,206 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:19,206 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:56:19,206 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:56:19,206 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:19,206 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:19,207 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:19,206 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:19,207 DEBUG [Thread-776] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:56:19,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:19,207 DEBUG [Thread-776] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:56:19,207 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:19,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:19,207 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:19,207 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:19,207 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:19,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:19,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:19,248 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=256 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/B/2b671d78ad1347eb9bf2906c128f1f8f 2023-03-21 11:56:19,268 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=256 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/C/e47c5662eca0464e8132fee808566f96 2023-03-21 11:56:19,273 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/A/bd20561c9bba4179b307acf694437b6c as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/bd20561c9bba4179b307acf694437b6c 2023-03-21 11:56:19,281 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/bd20561c9bba4179b307acf694437b6c, entries=150, sequenceid=256, filesize=25.3 K 2023-03-21 11:56:19,282 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/B/2b671d78ad1347eb9bf2906c128f1f8f as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/2b671d78ad1347eb9bf2906c128f1f8f 2023-03-21 11:56:19,286 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/2b671d78ad1347eb9bf2906c128f1f8f, entries=150, sequenceid=256, filesize=11.8 K 2023-03-21 11:56:19,290 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/C/e47c5662eca0464e8132fee808566f96 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/e47c5662eca0464e8132fee808566f96 2023-03-21 11:56:19,294 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/e47c5662eca0464e8132fee808566f96, entries=150, sequenceid=256, filesize=11.8 K 2023-03-21 11:56:19,295 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~64.85 KB/66410, heapSize ~175.11 KB/179312, currentSize=114.05 KB/116790 for 8041406d0a1ddc5b9409edb79457d352 in 924ms, sequenceid=256, compaction requested=true 2023-03-21 11:56:19,295 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8041406d0a1ddc5b9409edb79457d352: 2023-03-21 11:56:19,295 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-21 11:56:19,295 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-21 11:56:19,295 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-21 11:56:19,295 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-21 11:56:19,295 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-21 11:56:19,303 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 96894 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-21 11:56:19,303 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 46058 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-21 11:56:19,303 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1912): 8041406d0a1ddc5b9409edb79457d352/A is initiating minor compaction (all files) 2023-03-21 11:56:19,303 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1912): 8041406d0a1ddc5b9409edb79457d352/B is initiating minor compaction (all files) 2023-03-21 11:56:19,303 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8041406d0a1ddc5b9409edb79457d352/A in TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. 2023-03-21 11:56:19,303 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8041406d0a1ddc5b9409edb79457d352/B in TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. 2023-03-21 11:56:19,303 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/273a452d436748db9b85a0a70c1d5ad1, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/ddd9bddd917c4463a3d0cec5fe0b0f26, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/73626e4eaa22461cb38d637952b66fb0, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/bd20561c9bba4179b307acf694437b6c] into tmpdir=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp, totalSize=94.6 K 2023-03-21 11:56:19,303 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/be0be94d0dba4e219c17a9369250d1f2, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/dc4187adff83447e9f945ac9ec074ce7, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/b9c74a7141ed44b990ae5731c4af571f, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/2b671d78ad1347eb9bf2906c128f1f8f] into tmpdir=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp, totalSize=45.0 K 2023-03-21 11:56:19,303 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting be0be94d0dba4e219c17a9369250d1f2, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=165, earliestPutTs=1679399776245 2023-03-21 11:56:19,303 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 273a452d436748db9b85a0a70c1d5ad1, keycount=150, bloomtype=ROW, size=25.6 K, encoding=NONE, compression=NONE, seqNum=165, earliestPutTs=1679399776245 2023-03-21 11:56:19,304 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting dc4187adff83447e9f945ac9ec074ce7, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=202, earliestPutTs=1679399776582 2023-03-21 11:56:19,304 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting ddd9bddd917c4463a3d0cec5fe0b0f26, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=202, earliestPutTs=1679399776582 2023-03-21 11:56:19,304 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting b9c74a7141ed44b990ae5731c4af571f, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=213, earliestPutTs=1679399777215 2023-03-21 11:56:19,304 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 73626e4eaa22461cb38d637952b66fb0, keycount=100, bloomtype=ROW, size=18.5 K, encoding=NONE, compression=NONE, seqNum=213, earliestPutTs=1679399777215 2023-03-21 11:56:19,305 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 2b671d78ad1347eb9bf2906c128f1f8f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=256, earliestPutTs=1679399778363 2023-03-21 11:56:19,305 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting bd20561c9bba4179b307acf694437b6c, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=256, earliestPutTs=1679399778363 2023-03-21 11:56:19,307 DEBUG [Thread-776] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:56:19,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:56:19,309 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:56:19,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:56:19,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:56:19,310 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:56:19,311 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:19,311 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:56:19,311 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:56:19,311 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:19,311 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:19,312 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:19,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:19,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:19,312 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:19,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:19,314 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:19,314 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:56:19,315 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:19,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:56:19,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:19,315 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:56:19,318 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:56:19,318 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:56:19,318 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:56:19,318 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:19,318 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:56:19,321 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:56:19,321 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:56:19,321 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. 2023-03-21 11:56:19,323 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] throttle.PressureAwareThroughputController(145): 8041406d0a1ddc5b9409edb79457d352#B#compaction#130 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-21 11:56:19,327 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. started... 2023-03-21 11:56:19,327 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8041406d0a1ddc5b9409edb79457d352 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-21 11:56:19,328 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8041406d0a1ddc5b9409edb79457d352, store=A 2023-03-21 11:56:19,328 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:56:19,328 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8041406d0a1ddc5b9409edb79457d352, store=B 2023-03-21 11:56:19,328 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:56:19,328 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8041406d0a1ddc5b9409edb79457d352, store=C 2023-03-21 11:56:19,328 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:56:19,332 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8041406d0a1ddc5b9409edb79457d352#A#compaction#129 average throughput is 1.31 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-21 11:56:19,361 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/B/d61116124a914c9994b96eedba8e4634 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/d61116124a914c9994b96eedba8e4634 2023-03-21 11:56:19,366 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 8041406d0a1ddc5b9409edb79457d352/B of 8041406d0a1ddc5b9409edb79457d352 into d61116124a914c9994b96eedba8e4634(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-21 11:56:19,366 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8041406d0a1ddc5b9409edb79457d352: 2023-03-21 11:56:19,366 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352., storeName=8041406d0a1ddc5b9409edb79457d352/B, priority=12, startTime=1679399779295; duration=0sec 2023-03-21 11:56:19,366 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-21 11:56:19,366 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-21 11:56:19,368 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032176f7842d065142f3ae36c66bcddeb3ab to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032176f7842d065142f3ae36c66bcddeb3ab 2023-03-21 11:56:19,368 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 46058 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-21 11:56:19,368 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1912): 8041406d0a1ddc5b9409edb79457d352/C is initiating minor compaction (all files) 2023-03-21 11:56:19,368 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8041406d0a1ddc5b9409edb79457d352/C in TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. 2023-03-21 11:56:19,368 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/f86db8c925ef4b43a1ab01df1e57e2df, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/ca9222ca5e6e4f3785f88820721dc327, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/27c6a44ae8264084af2ab9acb81dc3e7, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/e47c5662eca0464e8132fee808566f96] into tmpdir=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp, totalSize=45.0 K 2023-03-21 11:56:19,369 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting f86db8c925ef4b43a1ab01df1e57e2df, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=165, earliestPutTs=1679399776245 2023-03-21 11:56:19,369 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting ca9222ca5e6e4f3785f88820721dc327, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=202, earliestPutTs=1679399776582 2023-03-21 11:56:19,369 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 27c6a44ae8264084af2ab9acb81dc3e7, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=213, earliestPutTs=1679399777215 2023-03-21 11:56:19,369 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting e47c5662eca0464e8132fee808566f96, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=256, earliestPutTs=1679399778363 2023-03-21 11:56:19,374 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=276, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/A/dc4edfa403fb45b9afa838a455f3b7ef 2023-03-21 11:56:19,378 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] throttle.PressureAwareThroughputController(145): 8041406d0a1ddc5b9409edb79457d352#C#compaction#132 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-21 11:56:19,400 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=276 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/B/1acd85607daa4a8b8d7dde1ef1c9e486 2023-03-21 11:56:19,412 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=276 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/C/9c7c65720183487f9af10bcf5e13f40a 2023-03-21 11:56:19,416 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/A/dc4edfa403fb45b9afa838a455f3b7ef as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/dc4edfa403fb45b9afa838a455f3b7ef 2023-03-21 11:56:19,425 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/dc4edfa403fb45b9afa838a455f3b7ef, entries=150, sequenceid=276, filesize=25.4 K 2023-03-21 11:56:19,428 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/B/1acd85607daa4a8b8d7dde1ef1c9e486 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/1acd85607daa4a8b8d7dde1ef1c9e486 2023-03-21 11:56:19,436 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/1acd85607daa4a8b8d7dde1ef1c9e486, entries=150, sequenceid=276, filesize=11.9 K 2023-03-21 11:56:19,451 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/C/9c7c65720183487f9af10bcf5e13f40a as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/9c7c65720183487f9af10bcf5e13f40a 2023-03-21 11:56:19,469 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/9c7c65720183487f9af10bcf5e13f40a, entries=150, sequenceid=276, filesize=11.9 K 2023-03-21 11:56:19,483 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=0 B/0 for 8041406d0a1ddc5b9409edb79457d352 in 143ms, sequenceid=276, compaction requested=false 2023-03-21 11:56:19,489 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8041406d0a1ddc5b9409edb79457d352: 2023-03-21 11:56:19,489 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. 2023-03-21 11:56:19,489 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:56:19,490 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:56:19,490 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:19,490 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:56:19,490 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:56:19,491 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:19,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:19,491 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:19,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:19,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:19,492 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:19,492 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:56:19,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:19,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:19,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:19,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:19,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:19,495 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:56:19,495 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@11502b49[Count = 0] remaining members to acquire global barrier 2023-03-21 11:56:19,495 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:56:19,495 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:19,496 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:19,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:19,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:19,496 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:56:19,496 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:56:19,496 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:56:19,496 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:19,496 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:56:19,497 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:19,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:19,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:19,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:19,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:19,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:19,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:19,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:19,499 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:56:19,499 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:19,499 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:56:19,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:19,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:19,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:19,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:56:19,502 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:56:19,502 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:56:19,502 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:56:19,502 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:56:19,502 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:56:19,502 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:56:19,506 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:19,506 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:19,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:19,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:19,506 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:19,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:19,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:19,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:19,514 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:19,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:19,514 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:19,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:19,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:19,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:19,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:19,516 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:19,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:19,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:19,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:19,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:19,518 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:19,518 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:19,518 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:19,518 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:19,518 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:19,518 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:19,518 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:19,519 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:56:19,519 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:56:19,518 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:19,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:19,518 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:19,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:19,519 DEBUG [Thread-776] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:56:19,519 DEBUG [Thread-776] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:56:19,519 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:19,519 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:56:19,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:19,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:19,569 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8041406d0a1ddc5b9409edb79457d352 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-21 11:56:19,569 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8041406d0a1ddc5b9409edb79457d352, store=A 2023-03-21 11:56:19,569 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:56:19,569 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8041406d0a1ddc5b9409edb79457d352, store=B 2023-03-21 11:56:19,569 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:56:19,569 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8041406d0a1ddc5b9409edb79457d352, store=C 2023-03-21 11:56:19,569 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:56:19,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(9158): Flush requested on 8041406d0a1ddc5b9409edb79457d352 2023-03-21 11:56:19,619 DEBUG [Thread-776] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:56:19,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:56:19,622 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:56:19,632 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=859, cellSize=39335, totalHeapSize=89576, min timestamp=1679399779572, max timestamp=1679399779631Num uniques 859; 2023-03-21 11:56:19,632 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=39335, suffix heap size=117064,new segment heap size=89576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=859 2023-03-21 11:56:19,632 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=859, cellSize=39335, totalHeapSize=89576, min timestamp=1679399779572, max timestamp=1679399779631Num uniques 859; 2023-03-21 11:56:19,632 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=39335, suffix heap size=117064,new segment heap size=89576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=859 2023-03-21 11:56:19,633 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=859, cellSize=39335, totalHeapSize=89576, min timestamp=1679399779572, max timestamp=1679399779631Num uniques 859; 2023-03-21 11:56:19,633 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=39335, suffix heap size=117064,new segment heap size=89576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=859 2023-03-21 11:56:19,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:56:19,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:56:19,638 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:56:19,638 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:19,638 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:56:19,638 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:56:19,639 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:19,639 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:19,640 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:19,640 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:19,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:19,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:19,640 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:19,640 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:56:19,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:19,640 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:19,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:56:19,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:19,641 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:56:19,649 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:19,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55216 deadline: 1679399839639, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:19,654 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:19,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55208 deadline: 1679399839643, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:19,658 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:19,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55188 deadline: 1679399839645, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:19,659 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:19,660 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:56:19,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55190 deadline: 1679399839647, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:19,662 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:56:19,662 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:56:19,662 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:19,662 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:56:19,662 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:19,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55180 deadline: 1679399839649, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:19,678 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:56:19,678 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:56:19,679 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. 2023-03-21 11:56:19,679 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. started... 2023-03-21 11:56:19,681 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. as already flushing 2023-03-21 11:56:19,681 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. 2023-03-21 11:56:19,681 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:56:19,681 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:56:19,681 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:19,681 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:56:19,681 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:56:19,682 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:19,682 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:19,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:19,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:19,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:19,682 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:19,682 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:56:19,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:19,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:19,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:19,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:19,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:19,685 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:56:19,685 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@456a279f[Count = 0] remaining members to acquire global barrier 2023-03-21 11:56:19,685 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:56:19,685 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:19,686 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:19,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:19,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:19,686 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:56:19,686 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:56:19,686 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:56:19,687 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:19,687 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:56:19,687 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:56:19,687 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:19,687 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:56:19,687 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:19,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:19,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:19,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:19,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:19,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:19,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:19,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:19,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:19,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:19,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:19,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:56:19,695 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:56:19,695 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:56:19,695 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:56:19,695 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:56:19,695 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:56:19,695 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:56:19,700 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:19,700 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:19,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:19,700 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:19,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:19,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:19,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:19,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:19,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:19,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:19,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:19,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:19,701 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:19,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:19,702 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:19,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:19,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:19,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:19,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:19,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:19,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:19,708 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:19,708 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:19,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:19,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:19,709 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:19,709 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:19,709 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:19,709 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:19,709 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:19,709 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:56:19,709 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:56:19,709 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:19,709 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:19,709 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:19,709 DEBUG [Thread-776] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:56:19,709 DEBUG [Thread-776] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:56:19,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:19,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:19,709 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:56:19,761 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:19,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55216 deadline: 1679399839751, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:19,764 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:19,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55208 deadline: 1679399839755, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:19,766 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:19,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55188 deadline: 1679399839760, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:19,773 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:19,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55190 deadline: 1679399839764, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:19,774 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:19,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55180 deadline: 1679399839764, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:19,799 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/C/353108bee7814c769e7b79546aafd405 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/353108bee7814c769e7b79546aafd405 2023-03-21 11:56:19,807 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 8041406d0a1ddc5b9409edb79457d352/C of 8041406d0a1ddc5b9409edb79457d352 into 353108bee7814c769e7b79546aafd405(size=12.2 K), total size for store is 24.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-21 11:56:19,807 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8041406d0a1ddc5b9409edb79457d352: 2023-03-21 11:56:19,807 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352., storeName=8041406d0a1ddc5b9409edb79457d352/C, priority=12, startTime=1679399779295; duration=0sec 2023-03-21 11:56:19,807 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-21 11:56:19,809 DEBUG [Thread-776] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:56:19,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:56:19,812 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:56:19,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:56:19,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:56:19,817 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:56:19,817 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:19,817 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:56:19,817 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:56:19,818 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:19,818 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:19,820 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:19,820 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:19,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:19,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:19,821 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:19,821 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:56:19,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:19,821 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:19,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:56:19,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:19,821 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:56:19,855 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:56:19,855 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:56:19,855 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:56:19,855 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:19,855 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:56:19,887 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:56:19,888 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:56:19,888 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. 2023-03-21 11:56:19,888 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. started... 2023-03-21 11:56:19,888 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. as already flushing 2023-03-21 11:56:19,888 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. 2023-03-21 11:56:19,888 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:56:19,888 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:56:19,888 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:19,888 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:56:19,888 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:56:19,889 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:19,889 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:19,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:19,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:19,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:19,890 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:19,890 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:56:19,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:19,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:19,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:19,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:19,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:19,891 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:56:19,891 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c432311[Count = 0] remaining members to acquire global barrier 2023-03-21 11:56:19,891 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:56:19,891 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:19,892 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:19,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:19,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:19,892 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:19,892 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:56:19,892 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:56:19,893 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:56:19,893 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:56:19,894 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:56:19,894 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:19,894 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:56:19,894 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:19,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:19,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:19,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:19,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:19,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:19,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:19,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:19,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:19,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:19,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:19,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:56:19,897 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:56:19,897 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:56:19,898 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:56:19,898 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:56:19,898 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:56:19,898 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:56:19,916 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:19,916 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:19,917 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:19,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:19,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:19,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:19,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:19,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:19,917 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:19,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:19,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:19,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:19,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:19,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:19,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:19,919 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:19,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:19,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:19,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:19,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:19,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:19,923 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:19,923 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:19,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:19,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:19,923 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:19,923 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:19,923 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:19,923 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:19,923 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:19,923 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:19,923 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:19,923 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:19,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:19,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:19,924 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:56:19,923 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:56:19,924 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:56:19,924 DEBUG [Thread-776] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:56:19,924 DEBUG [Thread-776] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:56:19,965 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:19,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55216 deadline: 1679399839964, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:19,969 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:19,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55208 deadline: 1679399839966, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:19,974 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:19,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55188 deadline: 1679399839968, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:19,982 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:19,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55190 deadline: 1679399839975, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:19,983 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:19,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55180 deadline: 1679399839975, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:20,024 DEBUG [Thread-776] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:56:20,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:56:20,026 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:56:20,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:56:20,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:56:20,030 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:56:20,030 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:20,030 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:56:20,030 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:56:20,031 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:20,031 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:20,031 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:20,031 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:20,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:20,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:20,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:20,032 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:20,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:56:20,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:20,032 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:56:20,033 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:20,033 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:56:20,046 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:56:20,046 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:56:20,046 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:56:20,046 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:20,046 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:56:20,049 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:56:20,049 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:56:20,065 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. 2023-03-21 11:56:20,065 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. started... 2023-03-21 11:56:20,065 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. as already flushing 2023-03-21 11:56:20,065 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. 2023-03-21 11:56:20,065 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:56:20,065 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:56:20,065 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:20,065 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:56:20,065 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:56:20,068 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:20,068 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:20,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:20,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:20,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:20,068 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:20,068 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:56:20,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:20,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:20,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:20,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:20,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:20,069 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:56:20,070 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6881a5d9[Count = 0] remaining members to acquire global barrier 2023-03-21 11:56:20,070 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:56:20,070 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:20,070 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:20,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:20,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:20,070 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:56:20,071 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:56:20,071 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:56:20,071 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:20,071 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:56:20,071 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:20,071 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:56:20,072 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:20,072 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:56:20,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:20,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:20,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:20,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:20,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:20,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:20,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:20,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:20,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:20,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:20,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:56:20,075 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:56:20,077 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:56:20,077 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:56:20,077 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:56:20,077 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:56:20,077 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:56:20,083 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:20,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:20,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:20,084 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:20,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:20,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:20,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:20,084 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:20,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:20,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:20,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:20,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:20,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:20,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:20,088 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:20,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:20,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:20,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:20,092 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:20,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:20,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:20,096 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230321132b6acb18d348eea41c6f007c10ff3f to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230321132b6acb18d348eea41c6f007c10ff3f 2023-03-21 11:56:20,100 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:20,100 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:20,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:20,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:20,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:20,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:20,105 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:20,105 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:20,105 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:20,105 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:20,105 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:20,105 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:20,105 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:20,106 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:20,106 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:56:20,106 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:56:20,107 DEBUG [Thread-776] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:56:20,107 DEBUG [Thread-776] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:56:20,207 DEBUG [Thread-776] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:56:20,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:56:20,209 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:56:20,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:56:20,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:56:20,227 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:56:20,227 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:20,227 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:56:20,227 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:56:20,228 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:20,228 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:20,229 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:20,229 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:20,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:20,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:20,229 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:20,229 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:56:20,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:20,230 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:20,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:56:20,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:20,230 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:56:20,242 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:56:20,242 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:56:20,242 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:56:20,242 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:20,242 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:56:20,245 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:56:20,245 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:56:20,245 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. 2023-03-21 11:56:20,245 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. started... 2023-03-21 11:56:20,245 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. as already flushing 2023-03-21 11:56:20,245 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. 2023-03-21 11:56:20,245 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:56:20,245 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:56:20,245 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:20,246 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:56:20,246 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:56:20,247 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:20,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:20,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:20,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:20,247 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:20,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:20,247 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:20,247 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:56:20,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:20,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:20,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:20,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:20,249 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:56:20,249 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@386ef891[Count = 0] remaining members to acquire global barrier 2023-03-21 11:56:20,249 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:56:20,249 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:20,249 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:20,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:20,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:20,249 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:56:20,249 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:56:20,250 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:56:20,250 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:20,250 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:56:20,251 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:20,251 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:56:20,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:20,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:20,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:20,251 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:20,251 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:56:20,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:20,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:20,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:20,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:20,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:20,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:20,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:20,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:56:20,253 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:56:20,253 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:56:20,253 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:56:20,253 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:56:20,253 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:56:20,253 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:56:20,255 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:20,255 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:20,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:20,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:20,255 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:20,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:20,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:20,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:20,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:20,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:20,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:20,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:20,256 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:20,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:20,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:20,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:20,257 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:20,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:20,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:20,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:20,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:20,259 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:20,259 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:20,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:20,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:20,260 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:20,260 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:20,260 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:20,260 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:20,260 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:20,260 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:20,260 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:20,260 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:20,260 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:56:20,260 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:56:20,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:20,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:20,260 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:56:20,260 DEBUG [Thread-776] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:56:20,260 DEBUG [Thread-776] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:56:20,271 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:20,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55216 deadline: 1679399840268, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:20,281 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:20,282 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:20,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55208 deadline: 1679399840276, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:20,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55188 deadline: 1679399840277, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:20,288 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:20,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55190 deadline: 1679399840285, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:20,289 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:20,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55180 deadline: 1679399840285, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:20,292 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/A/db8ceeba926a496eb0d57baecd82fe7f as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/db8ceeba926a496eb0d57baecd82fe7f 2023-03-21 11:56:20,305 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 8041406d0a1ddc5b9409edb79457d352/A of 8041406d0a1ddc5b9409edb79457d352 into db8ceeba926a496eb0d57baecd82fe7f(size=25.7 K), total size for store is 51.2 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-03-21 11:56:20,305 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8041406d0a1ddc5b9409edb79457d352: 2023-03-21 11:56:20,305 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352., storeName=8041406d0a1ddc5b9409edb79457d352/A, priority=12, startTime=1679399779295; duration=1sec 2023-03-21 11:56:20,306 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-21 11:56:20,360 DEBUG [Thread-776] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:56:20,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:56:20,363 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:56:20,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:56:20,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:56:20,366 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:56:20,366 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:20,366 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:56:20,366 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:56:20,366 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:20,366 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:20,367 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:20,367 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:20,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:20,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:20,367 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:20,367 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:56:20,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:20,367 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:20,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:56:20,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:20,368 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:56:20,368 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:56:20,368 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:56:20,368 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:56:20,368 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:20,368 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:56:20,369 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:56:20,369 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:56:20,369 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. 2023-03-21 11:56:20,369 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. started... 2023-03-21 11:56:20,369 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. as already flushing 2023-03-21 11:56:20,369 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. 2023-03-21 11:56:20,373 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:56:20,373 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:56:20,373 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:20,373 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:56:20,373 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:56:20,382 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:20,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:20,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:20,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:20,383 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:20,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:20,383 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:20,383 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:56:20,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:20,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:20,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:20,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:20,384 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:56:20,384 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:56:20,384 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1eac5ceb[Count = 0] remaining members to acquire global barrier 2023-03-21 11:56:20,384 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:20,385 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:20,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:20,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:20,385 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:56:20,385 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:56:20,385 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:20,385 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:56:20,385 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:56:20,386 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:56:20,386 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:20,386 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:20,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:20,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:20,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:20,386 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:56:20,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:20,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:20,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:20,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:20,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:20,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:20,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:20,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:56:20,392 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:56:20,392 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:56:20,392 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:56:20,392 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:56:20,392 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:56:20,392 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:56:20,398 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:20,398 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:20,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:20,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:20,399 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:20,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:20,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:20,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:20,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:20,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:20,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:20,399 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:20,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:20,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:20,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:20,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:20,400 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:20,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:20,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:20,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:20,402 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:20,402 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:56:20,402 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:56:20,402 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:20,402 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:20,402 DEBUG [Thread-776] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:56:20,403 DEBUG [Thread-776] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:56:20,402 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:20,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:20,402 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:20,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:20,403 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:20,403 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:20,403 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:20,403 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:20,403 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:20,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:20,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:20,403 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:56:20,503 DEBUG [Thread-776] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:56:20,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:56:20,504 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:56:20,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:56:20,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:56:20,505 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:56:20,505 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:20,506 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:56:20,506 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:56:20,506 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:20,506 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:20,506 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:20,506 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:20,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:20,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:20,507 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:20,507 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:56:20,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:20,507 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:20,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:56:20,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:20,507 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:56:20,508 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:56:20,508 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:56:20,508 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:56:20,508 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:20,508 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:56:20,508 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:56:20,508 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:56:20,508 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. 2023-03-21 11:56:20,508 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. started... 2023-03-21 11:56:20,508 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. as already flushing 2023-03-21 11:56:20,508 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. 2023-03-21 11:56:20,508 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:56:20,508 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:56:20,508 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:20,508 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:56:20,508 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:56:20,509 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:20,509 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:20,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:20,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:20,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:20,509 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:20,509 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:56:20,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:20,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:20,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:20,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:20,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:20,510 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:56:20,511 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:56:20,511 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b92bf84[Count = 0] remaining members to acquire global barrier 2023-03-21 11:56:20,511 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:20,511 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:20,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:20,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:20,511 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:56:20,511 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:56:20,511 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:56:20,511 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:20,511 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:56:20,512 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:20,512 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:56:20,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:20,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:20,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:20,512 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:20,512 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:56:20,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:20,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:20,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:20,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:20,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:20,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:20,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:20,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:56:20,514 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:56:20,514 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:56:20,514 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:56:20,514 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:56:20,514 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:56:20,514 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:56:20,515 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:20,515 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:20,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:20,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:20,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:20,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:20,515 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:20,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:20,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:20,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:20,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:20,515 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:20,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:20,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:20,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:20,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:20,516 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:20,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:20,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:20,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:20,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:20,518 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:20,518 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:20,518 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:20,518 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:20,518 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:56:20,518 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:56:20,518 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:20,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:20,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:20,518 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:56:20,518 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:20,518 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:20,518 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:20,518 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:20,518 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:20,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:20,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:20,522 DEBUG [Thread-776] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:56:20,522 DEBUG [Thread-776] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:56:20,544 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=287, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/A/cc333ece122d4659b167b753dde00f69 2023-03-21 11:56:20,622 DEBUG [Thread-776] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:56:20,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:56:20,624 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:56:20,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:56:20,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:56:20,625 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:56:20,625 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:20,625 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:56:20,625 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:56:20,625 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:20,625 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:20,626 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:20,626 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:20,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:20,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:20,627 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:20,627 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:56:20,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:20,627 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:20,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:56:20,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:20,629 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:56:20,629 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:56:20,630 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:56:20,630 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:56:20,630 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:20,630 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:56:20,630 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:56:20,630 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:56:20,630 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. 2023-03-21 11:56:20,630 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. started... 2023-03-21 11:56:20,630 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. as already flushing 2023-03-21 11:56:20,630 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. 2023-03-21 11:56:20,630 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:56:20,630 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:56:20,630 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:20,630 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:56:20,630 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:56:20,639 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:20,639 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:20,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:20,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:20,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:20,640 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:20,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:20,640 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:56:20,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:20,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:20,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:20,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:20,641 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:56:20,641 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1ed86242[Count = 0] remaining members to acquire global barrier 2023-03-21 11:56:20,641 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:56:20,641 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:20,641 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:20,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:20,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:20,642 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:56:20,642 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:56:20,642 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:20,642 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:56:20,642 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:56:20,642 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:20,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:20,642 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:56:20,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:20,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:20,642 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:20,643 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:56:20,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:20,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:20,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:20,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:20,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:20,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:20,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:20,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:56:20,644 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:56:20,644 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:56:20,644 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:56:20,644 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:56:20,644 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:56:20,644 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:56:20,645 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:20,645 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:20,645 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:20,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:20,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:20,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:20,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:20,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:20,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:20,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:20,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:20,646 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:20,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:20,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:20,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:20,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:20,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:20,647 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:20,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:20,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:20,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:20,648 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:20,648 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:20,648 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:20,648 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:20,648 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:20,648 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:20,648 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:56:20,648 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:56:20,648 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:20,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:20,648 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:20,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:20,648 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:20,649 DEBUG [Thread-776] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:56:20,649 DEBUG [Thread-776] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:56:20,649 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:20,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:20,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:20,749 DEBUG [Thread-776] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:56:20,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:56:20,751 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:56:20,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:56:20,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:56:20,752 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:56:20,752 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:20,752 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:56:20,752 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:56:20,752 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:20,752 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:20,753 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:20,753 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:20,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:20,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:20,753 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:20,753 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:56:20,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:20,753 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:20,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:56:20,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:20,753 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:56:20,754 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:56:20,754 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:56:20,754 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:56:20,754 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:20,754 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:56:20,754 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:56:20,754 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:56:20,754 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. 2023-03-21 11:56:20,754 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. started... 2023-03-21 11:56:20,754 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. as already flushing 2023-03-21 11:56:20,754 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. 2023-03-21 11:56:20,754 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:56:20,754 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:56:20,754 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:20,754 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:56:20,754 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:56:20,755 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:20,755 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:20,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:20,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:20,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:20,755 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:20,755 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:56:20,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:20,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:20,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:20,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:20,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:20,756 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:56:20,757 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:56:20,757 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@513dbb78[Count = 0] remaining members to acquire global barrier 2023-03-21 11:56:20,757 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:20,757 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:20,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:20,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:20,757 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:56:20,757 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:20,757 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:56:20,757 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:56:20,757 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:56:20,759 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:56:20,759 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:20,759 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:20,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:20,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:20,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:20,759 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:56:20,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:20,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:20,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:20,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:20,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:20,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:20,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:20,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:56:20,760 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:56:20,760 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:56:20,760 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:56:20,760 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:56:20,760 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:56:20,761 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:56:20,761 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:20,761 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:20,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:20,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:20,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:20,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:20,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:20,761 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:20,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:20,762 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:20,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:20,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:20,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:20,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:20,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:20,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:20,763 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:20,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:20,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:20,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:20,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:20,764 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:20,764 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:20,764 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:20,764 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:20,764 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:20,764 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:20,764 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:20,764 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:56:20,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:20,764 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:56:20,764 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:20,764 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:56:20,764 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:20,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:20,764 DEBUG [Thread-776] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:56:20,764 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:20,764 DEBUG [Thread-776] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:56:20,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:20,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:20,778 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:20,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55216 deadline: 1679399840777, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:20,787 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:20,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55208 deadline: 1679399840785, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:20,788 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:20,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55188 deadline: 1679399840786, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:20,793 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:20,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55190 deadline: 1679399840791, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:20,795 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:20,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55180 deadline: 1679399840794, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:20,864 DEBUG [Thread-776] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:56:20,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:56:20,866 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:56:20,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:56:20,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:56:20,867 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:56:20,867 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:20,867 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:56:20,867 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:56:20,868 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:20,868 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:20,868 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:20,868 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:20,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:20,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:20,869 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:20,869 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:56:20,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:20,869 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:20,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:56:20,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:20,869 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:56:20,869 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:56:20,869 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:56:20,870 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:56:20,870 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:20,870 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:56:20,870 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:56:20,870 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:56:20,870 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. 2023-03-21 11:56:20,870 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. started... 2023-03-21 11:56:20,870 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. as already flushing 2023-03-21 11:56:20,870 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. 2023-03-21 11:56:20,870 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:56:20,870 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:56:20,870 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:20,870 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:56:20,870 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:56:20,871 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:20,871 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:20,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:20,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:20,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:20,871 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:20,871 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:56:20,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:20,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:20,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:20,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:20,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:20,872 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:56:20,872 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d8afeb0[Count = 0] remaining members to acquire global barrier 2023-03-21 11:56:20,872 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:56:20,872 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:20,873 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:20,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:20,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:20,873 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:56:20,873 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:56:20,873 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:56:20,873 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:20,873 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:56:20,874 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:20,874 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:56:20,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:20,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:20,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:20,874 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:20,874 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:56:20,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:20,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:20,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:20,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:20,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:20,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:20,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:20,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:56:20,876 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:56:20,876 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:56:20,876 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:56:20,876 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:56:20,876 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:56:20,876 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:56:20,877 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:20,877 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:20,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:20,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:20,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:20,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:20,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:20,877 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:20,877 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:20,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:20,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:20,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:20,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:20,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:20,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:20,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:20,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:20,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:20,879 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:20,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:20,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:20,880 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:20,880 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:20,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:20,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:20,880 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:20,880 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:56:20,880 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:56:20,880 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:20,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:20,880 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:20,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:20,880 DEBUG [Thread-776] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:56:20,880 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:20,880 DEBUG [Thread-776] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:56:20,880 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:20,880 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:20,880 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:20,880 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:20,959 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=287 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/B/c23741cb63914d0c92ab2b352daf8299 2023-03-21 11:56:20,980 DEBUG [Thread-776] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:56:20,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:56:20,982 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:56:20,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:56:20,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:56:20,995 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=287 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/C/d5f8bdf4efde44c5956a0939d1a1c85b 2023-03-21 11:56:20,995 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:56:20,995 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:20,995 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:56:20,995 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:56:20,995 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:20,995 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:20,996 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:20,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:20,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:20,996 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:20,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:20,997 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:20,997 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:56:20,997 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:20,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:56:20,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:20,997 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:56:21,000 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/A/cc333ece122d4659b167b753dde00f69 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/cc333ece122d4659b167b753dde00f69 2023-03-21 11:56:21,000 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:56:21,000 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:56:21,000 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:56:21,000 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:21,000 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:56:21,006 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:56:21,006 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. 2023-03-21 11:56:21,006 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:56:21,006 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. started... 2023-03-21 11:56:21,006 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. as already flushing 2023-03-21 11:56:21,006 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. 2023-03-21 11:56:21,006 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:56:21,006 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:56:21,006 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:21,007 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:56:21,007 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:56:21,016 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/cc333ece122d4659b167b753dde00f69, entries=350, sequenceid=287, filesize=52.9 K 2023-03-21 11:56:21,016 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:21,016 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:21,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:21,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:21,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:21,017 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:21,017 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:56:21,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:21,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:21,017 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/B/c23741cb63914d0c92ab2b352daf8299 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/c23741cb63914d0c92ab2b352daf8299 2023-03-21 11:56:21,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:21,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:21,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:21,019 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:56:21,019 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@438a1a1a[Count = 0] remaining members to acquire global barrier 2023-03-21 11:56:21,019 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:56:21,019 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:21,023 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:21,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:21,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:21,023 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:56:21,023 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:21,023 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:56:21,023 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:56:21,023 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:56:21,024 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:56:21,024 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:21,024 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:21,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:21,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:21,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:21,024 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:56:21,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:21,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:21,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:21,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:21,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:21,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:21,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:21,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:56:21,026 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:56:21,026 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:56:21,026 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:56:21,026 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:56:21,026 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:56:21,026 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:56:21,027 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:21,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:21,027 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:21,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:21,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:21,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:21,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:21,027 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:21,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:21,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:21,027 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:21,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:21,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:21,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:21,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:21,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:21,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:21,028 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:21,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:21,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:21,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:21,029 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:21,029 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:21,029 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:21,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:21,029 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:21,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:21,029 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:56:21,029 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:56:21,029 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:21,029 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:21,029 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:21,029 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:21,029 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:21,029 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:21,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:21,029 DEBUG [Thread-776] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:56:21,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:21,030 DEBUG [Thread-776] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:56:21,032 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/c23741cb63914d0c92ab2b352daf8299, entries=150, sequenceid=287, filesize=11.9 K 2023-03-21 11:56:21,034 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/C/d5f8bdf4efde44c5956a0939d1a1c85b as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/d5f8bdf4efde44c5956a0939d1a1c85b 2023-03-21 11:56:21,038 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/d5f8bdf4efde44c5956a0939d1a1c85b, entries=150, sequenceid=287, filesize=11.9 K 2023-03-21 11:56:21,039 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=154.31 KB/158010 for 8041406d0a1ddc5b9409edb79457d352 in 1471ms, sequenceid=287, compaction requested=true 2023-03-21 11:56:21,039 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8041406d0a1ddc5b9409edb79457d352: 2023-03-21 11:56:21,039 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-21 11:56:21,039 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-21 11:56:21,039 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-21 11:56:21,039 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-21 11:56:21,039 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-21 11:56:21,040 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 106600 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-21 11:56:21,040 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36852 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-21 11:56:21,040 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1912): 8041406d0a1ddc5b9409edb79457d352/A is initiating minor compaction (all files) 2023-03-21 11:56:21,040 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1912): 8041406d0a1ddc5b9409edb79457d352/B is initiating minor compaction (all files) 2023-03-21 11:56:21,040 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8041406d0a1ddc5b9409edb79457d352/A in TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. 2023-03-21 11:56:21,040 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8041406d0a1ddc5b9409edb79457d352/B in TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. 2023-03-21 11:56:21,040 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/d61116124a914c9994b96eedba8e4634, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/1acd85607daa4a8b8d7dde1ef1c9e486, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/c23741cb63914d0c92ab2b352daf8299] into tmpdir=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp, totalSize=36.0 K 2023-03-21 11:56:21,040 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/db8ceeba926a496eb0d57baecd82fe7f, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/dc4edfa403fb45b9afa838a455f3b7ef, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/cc333ece122d4659b167b753dde00f69] into tmpdir=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp, totalSize=104.1 K 2023-03-21 11:56:21,041 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting d61116124a914c9994b96eedba8e4634, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=256, earliestPutTs=1679399778363 2023-03-21 11:56:21,041 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting db8ceeba926a496eb0d57baecd82fe7f, keycount=150, bloomtype=ROW, size=25.7 K, encoding=NONE, compression=NONE, seqNum=256, earliestPutTs=1679399778363 2023-03-21 11:56:21,041 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 1acd85607daa4a8b8d7dde1ef1c9e486, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=276, earliestPutTs=1679399778386 2023-03-21 11:56:21,041 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting dc4edfa403fb45b9afa838a455f3b7ef, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=276, earliestPutTs=1679399778386 2023-03-21 11:56:21,042 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting c23741cb63914d0c92ab2b352daf8299, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=287, earliestPutTs=1679399779550 2023-03-21 11:56:21,042 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting cc333ece122d4659b167b753dde00f69, keycount=350, bloomtype=ROW, size=52.9 K, encoding=NONE, compression=NONE, seqNum=287, earliestPutTs=1679399779542 2023-03-21 11:56:21,048 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] throttle.PressureAwareThroughputController(145): 8041406d0a1ddc5b9409edb79457d352#B#compaction#139 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-21 11:56:21,060 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8041406d0a1ddc5b9409edb79457d352#A#compaction#138 average throughput is 1.41 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-21 11:56:21,130 DEBUG [Thread-776] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:56:21,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:56:21,132 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:56:21,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:56:21,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:56:21,133 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:56:21,133 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:21,133 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:56:21,133 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:56:21,133 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:21,133 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:21,134 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:21,134 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:21,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:21,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:21,134 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:21,134 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:56:21,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:21,134 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:21,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:56:21,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:21,135 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:56:21,135 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:56:21,135 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:56:21,135 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:56:21,135 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:21,135 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:56:21,135 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:56:21,135 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:56:21,135 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. 2023-03-21 11:56:21,135 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. started... 2023-03-21 11:56:21,135 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8041406d0a1ddc5b9409edb79457d352 3/3 column families, dataSize=154.31 KB heapSize=379.13 KB 2023-03-21 11:56:21,135 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8041406d0a1ddc5b9409edb79457d352, store=A 2023-03-21 11:56:21,135 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-21 11:56:21,136 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8041406d0a1ddc5b9409edb79457d352, store=B 2023-03-21 11:56:21,136 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-21 11:56:21,136 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8041406d0a1ddc5b9409edb79457d352, store=C 2023-03-21 11:56:21,136 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-21 11:56:21,154 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032109130aa3f9df4172b2fc8ce6f9dae646 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032109130aa3f9df4172b2fc8ce6f9dae646 2023-03-21 11:56:21,466 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/B/5bf29df8ddd84de8bc87ca4ce1aa0d71 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/5bf29df8ddd84de8bc87ca4ce1aa0d71 2023-03-21 11:56:21,474 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8041406d0a1ddc5b9409edb79457d352/B of 8041406d0a1ddc5b9409edb79457d352 into 5bf29df8ddd84de8bc87ca4ce1aa0d71(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-21 11:56:21,474 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8041406d0a1ddc5b9409edb79457d352: 2023-03-21 11:56:21,474 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352., storeName=8041406d0a1ddc5b9409edb79457d352/B, priority=13, startTime=1679399781039; duration=0sec 2023-03-21 11:56:21,474 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-21 11:56:21,474 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-21 11:56:21,475 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36852 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-21 11:56:21,475 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1912): 8041406d0a1ddc5b9409edb79457d352/C is initiating minor compaction (all files) 2023-03-21 11:56:21,476 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8041406d0a1ddc5b9409edb79457d352/C in TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. 2023-03-21 11:56:21,476 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/353108bee7814c769e7b79546aafd405, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/9c7c65720183487f9af10bcf5e13f40a, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/d5f8bdf4efde44c5956a0939d1a1c85b] into tmpdir=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp, totalSize=36.0 K 2023-03-21 11:56:21,476 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 353108bee7814c769e7b79546aafd405, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=256, earliestPutTs=1679399778363 2023-03-21 11:56:21,476 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 9c7c65720183487f9af10bcf5e13f40a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=276, earliestPutTs=1679399778386 2023-03-21 11:56:21,477 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting d5f8bdf4efde44c5956a0939d1a1c85b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=287, earliestPutTs=1679399779550 2023-03-21 11:56:21,492 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] throttle.PressureAwareThroughputController(145): 8041406d0a1ddc5b9409edb79457d352#C#compaction#141 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-21 11:56:21,562 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=315, memsize=51.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/A/f00401a98c504169bc10f4f32bb3979b 2023-03-21 11:56:21,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(9158): Flush requested on 8041406d0a1ddc5b9409edb79457d352 2023-03-21 11:56:21,781 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. as already flushing 2023-03-21 11:56:21,794 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:21,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55188 deadline: 1679399841791, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:21,795 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:21,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55216 deadline: 1679399841794, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:21,799 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:21,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55208 deadline: 1679399841795, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:21,799 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:21,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55180 deadline: 1679399841797, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:21,804 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:21,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55190 deadline: 1679399841803, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:21,898 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:21,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55216 deadline: 1679399841896, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:21,902 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:21,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55208 deadline: 1679399841899, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:21,906 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/C/5a814a018ec7427cab20a2fd2e327aa5 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/5a814a018ec7427cab20a2fd2e327aa5 2023-03-21 11:56:21,911 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8041406d0a1ddc5b9409edb79457d352/C of 8041406d0a1ddc5b9409edb79457d352 into 5a814a018ec7427cab20a2fd2e327aa5(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-21 11:56:21,911 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8041406d0a1ddc5b9409edb79457d352: 2023-03-21 11:56:21,911 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352., storeName=8041406d0a1ddc5b9409edb79457d352/C, priority=13, startTime=1679399781039; duration=0sec 2023-03-21 11:56:21,911 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-21 11:56:21,980 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=315 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/B/7a8f30c454494ff2b803cb440c4d4180 2023-03-21 11:56:22,106 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:22,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55216 deadline: 1679399842100, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:22,106 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:22,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55208 deadline: 1679399842104, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:22,298 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/A/33306f14338d42e5921bbcaefbdaf412 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/33306f14338d42e5921bbcaefbdaf412 2023-03-21 11:56:22,303 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8041406d0a1ddc5b9409edb79457d352/A of 8041406d0a1ddc5b9409edb79457d352 into 33306f14338d42e5921bbcaefbdaf412(size=26.0 K), total size for store is 26.0 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-03-21 11:56:22,303 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8041406d0a1ddc5b9409edb79457d352: 2023-03-21 11:56:22,303 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352., storeName=8041406d0a1ddc5b9409edb79457d352/A, priority=13, startTime=1679399781039; duration=1sec 2023-03-21 11:56:22,303 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-21 11:56:22,391 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=315 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/C/912fcc93b4ba433e946c283da6ac814c 2023-03-21 11:56:22,399 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/A/f00401a98c504169bc10f4f32bb3979b as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/f00401a98c504169bc10f4f32bb3979b 2023-03-21 11:56:22,409 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/f00401a98c504169bc10f4f32bb3979b, entries=150, sequenceid=315, filesize=25.4 K 2023-03-21 11:56:22,410 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/B/7a8f30c454494ff2b803cb440c4d4180 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/7a8f30c454494ff2b803cb440c4d4180 2023-03-21 11:56:22,411 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:22,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55208 deadline: 1679399842410, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:22,413 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:22,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55216 deadline: 1679399842411, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:22,424 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/7a8f30c454494ff2b803cb440c4d4180, entries=150, sequenceid=315, filesize=11.9 K 2023-03-21 11:56:22,426 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/C/912fcc93b4ba433e946c283da6ac814c as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/912fcc93b4ba433e946c283da6ac814c 2023-03-21 11:56:22,454 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/912fcc93b4ba433e946c283da6ac814c, entries=150, sequenceid=315, filesize=11.9 K 2023-03-21 11:56:22,475 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~154.31 KB/158010, heapSize ~379.08 KB/388176, currentSize=53.67 KB/54960 for 8041406d0a1ddc5b9409edb79457d352 in 1340ms, sequenceid=315, compaction requested=false 2023-03-21 11:56:22,476 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8041406d0a1ddc5b9409edb79457d352: 2023-03-21 11:56:22,476 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. 2023-03-21 11:56:22,477 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:56:22,477 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:56:22,477 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:22,477 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:56:22,477 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:56:22,487 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:22,487 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:22,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:22,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:22,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:22,487 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:22,487 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:56:22,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:22,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:22,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:22,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:22,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:22,489 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:56:22,489 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e9dba5b[Count = 0] remaining members to acquire global barrier 2023-03-21 11:56:22,489 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:56:22,489 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:22,490 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:22,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:22,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:22,490 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:56:22,490 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:56:22,490 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:56:22,491 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:22,491 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:56:22,492 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:22,492 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:56:22,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:22,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:22,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:22,492 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:22,492 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:56:22,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:22,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:22,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:22,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:22,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:22,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:22,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:22,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:56:22,518 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:56:22,519 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:56:22,519 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:56:22,519 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:56:22,519 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:56:22,519 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:56:22,524 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:22,524 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:22,524 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:22,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:22,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:22,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:22,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:22,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:22,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:22,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:22,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:22,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:22,528 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:22,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:22,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:22,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:22,531 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:22,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:22,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:22,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:22,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:22,540 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:22,540 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:22,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:22,540 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:56:22,540 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:56:22,540 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:22,540 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:22,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:22,541 DEBUG [Thread-776] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:56:22,541 DEBUG [Thread-776] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:56:22,541 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:56:22,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:22,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:22,541 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:22,541 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:22,541 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:22,542 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:22,542 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:22,542 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:22,641 DEBUG [Thread-776] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:56:22,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:56:22,645 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:56:22,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:56:22,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:56:22,656 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:56:22,656 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:22,656 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:56:22,656 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:56:22,657 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:22,657 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:22,659 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:22,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:22,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:22,659 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:22,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:22,660 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:22,661 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:22,661 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:56:22,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:56:22,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:22,661 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:56:22,661 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:56:22,661 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:56:22,661 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:56:22,661 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:22,661 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:56:22,662 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:56:22,662 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:56:22,662 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. 2023-03-21 11:56:22,662 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. started... 2023-03-21 11:56:22,662 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8041406d0a1ddc5b9409edb79457d352 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-21 11:56:22,662 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8041406d0a1ddc5b9409edb79457d352, store=A 2023-03-21 11:56:22,662 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:56:22,662 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8041406d0a1ddc5b9409edb79457d352, store=B 2023-03-21 11:56:22,662 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:56:22,662 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8041406d0a1ddc5b9409edb79457d352, store=C 2023-03-21 11:56:22,662 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:56:22,755 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230321a6ef5802d6f547edaae1b691bc018c22 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230321a6ef5802d6f547edaae1b691bc018c22 2023-03-21 11:56:22,919 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. as already flushing 2023-03-21 11:56:22,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(9158): Flush requested on 8041406d0a1ddc5b9409edb79457d352 2023-03-21 11:56:22,985 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679399782926, max timestamp=1679399782980Num uniques 859; 2023-03-21 11:56:22,985 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-21 11:56:22,985 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679399782926, max timestamp=1679399782980Num uniques 859; 2023-03-21 11:56:22,985 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-21 11:56:22,987 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679399782926, max timestamp=1679399782980Num uniques 859; 2023-03-21 11:56:22,987 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-21 11:56:23,036 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:23,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55216 deadline: 1679399843033, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:23,037 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:23,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55208 deadline: 1679399843033, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:23,140 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:23,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55216 deadline: 1679399843138, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:23,141 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:23,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55208 deadline: 1679399843138, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:23,209 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=329, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/A/a0fe0dc5e09b4490b1492cfce690fcd3 2023-03-21 11:56:23,343 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:23,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55216 deadline: 1679399843342, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:23,344 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:23,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55208 deadline: 1679399843343, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:23,625 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=329 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/B/41e3b24dd07644448be2c272436db052 2023-03-21 11:56:23,647 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:23,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55216 deadline: 1679399843645, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:23,648 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:23,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55208 deadline: 1679399843646, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:23,799 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:23,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55188 deadline: 1679399843798, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:23,801 DEBUG [Thread-770] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4154 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352., hostname=jenkins-hbase20.apache.org,34369,1679399732030, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-21 11:56:23,807 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:23,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55180 deadline: 1679399843806, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:23,809 DEBUG [Thread-768] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4159 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352., hostname=jenkins-hbase20.apache.org,34369,1679399732030, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-21 11:56:23,820 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:23,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55190 deadline: 1679399843818, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:23,823 DEBUG [Thread-772] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4174 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352., hostname=jenkins-hbase20.apache.org,34369,1679399732030, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-21 11:56:24,039 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=329 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/C/70176395a5da41b2aca0a62c1e7f224b 2023-03-21 11:56:24,043 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/A/a0fe0dc5e09b4490b1492cfce690fcd3 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/a0fe0dc5e09b4490b1492cfce690fcd3 2023-03-21 11:56:24,047 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/a0fe0dc5e09b4490b1492cfce690fcd3, entries=150, sequenceid=329, filesize=25.4 K 2023-03-21 11:56:24,048 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/B/41e3b24dd07644448be2c272436db052 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/41e3b24dd07644448be2c272436db052 2023-03-21 11:56:24,052 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/41e3b24dd07644448be2c272436db052, entries=150, sequenceid=329, filesize=11.9 K 2023-03-21 11:56:24,053 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/C/70176395a5da41b2aca0a62c1e7f224b as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/70176395a5da41b2aca0a62c1e7f224b 2023-03-21 11:56:24,057 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/70176395a5da41b2aca0a62c1e7f224b, entries=150, sequenceid=329, filesize=11.9 K 2023-03-21 11:56:24,058 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 8041406d0a1ddc5b9409edb79457d352 in 1396ms, sequenceid=329, compaction requested=true 2023-03-21 11:56:24,058 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8041406d0a1ddc5b9409edb79457d352: 2023-03-21 11:56:24,058 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. 2023-03-21 11:56:24,058 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:56:24,058 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:56:24,058 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:24,058 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:56:24,058 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:56:24,059 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:24,059 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:24,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:24,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:24,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:24,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:24,060 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:24,060 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:56:24,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:24,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:24,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:24,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:24,061 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:56:24,061 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@22c7248b[Count = 0] remaining members to acquire global barrier 2023-03-21 11:56:24,061 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:56:24,061 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:24,062 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:24,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:24,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:24,062 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:56:24,062 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:24,062 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:56:24,062 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:56:24,062 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:56:24,065 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:56:24,065 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:24,065 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:56:24,065 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:24,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:24,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:24,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:24,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:24,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:24,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:24,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:24,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:24,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:24,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:24,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:56:24,069 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:56:24,069 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:56:24,069 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:56:24,069 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:56:24,069 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:56:24,070 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:56:24,073 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:24,073 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:24,073 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:24,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:24,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:24,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:24,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:24,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:24,073 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:24,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:24,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:24,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:24,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:24,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:24,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:24,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:24,074 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:24,076 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:24,076 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:24,076 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:24,076 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:24,076 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:24,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:24,076 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:24,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:24,076 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:56:24,077 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:56:24,076 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:24,077 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:24,077 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:24,077 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:24,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:24,077 DEBUG [Thread-776] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:56:24,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:24,077 DEBUG [Thread-776] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:56:24,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:24,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:24,078 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:56:24,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(9158): Flush requested on 8041406d0a1ddc5b9409edb79457d352 2023-03-21 11:56:24,154 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8041406d0a1ddc5b9409edb79457d352 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-03-21 11:56:24,154 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8041406d0a1ddc5b9409edb79457d352, store=A 2023-03-21 11:56:24,154 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-21 11:56:24,155 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8041406d0a1ddc5b9409edb79457d352, store=B 2023-03-21 11:56:24,155 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-21 11:56:24,155 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8041406d0a1ddc5b9409edb79457d352, store=C 2023-03-21 11:56:24,155 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-21 11:56:24,166 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:24,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55216 deadline: 1679399844164, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:24,169 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:24,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55208 deadline: 1679399844165, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:24,177 DEBUG [Thread-776] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:56:24,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:56:24,179 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:56:24,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:56:24,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:56:24,184 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:56:24,184 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:24,184 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:56:24,184 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:56:24,184 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:24,184 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:24,185 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:24,185 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:24,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:24,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:24,185 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:24,185 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:56:24,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:24,185 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:24,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:56:24,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:24,186 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:56:24,186 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:56:24,186 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:56:24,186 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:56:24,186 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:24,186 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:56:24,186 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:56:24,186 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:56:24,186 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. 2023-03-21 11:56:24,186 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. started... 2023-03-21 11:56:24,186 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. as already flushing 2023-03-21 11:56:24,186 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. 2023-03-21 11:56:24,186 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:56:24,186 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:56:24,186 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:24,186 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:56:24,186 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:56:24,187 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:24,187 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:24,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:24,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:24,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:24,187 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:24,188 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:56:24,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:24,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:24,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:24,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:24,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:24,188 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:56:24,189 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:56:24,189 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5842311d[Count = 0] remaining members to acquire global barrier 2023-03-21 11:56:24,189 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:24,189 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:24,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:24,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:24,189 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:56:24,189 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:56:24,189 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:24,189 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:56:24,189 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:56:24,190 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:56:24,190 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:24,190 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:24,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:24,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:24,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:24,190 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:56:24,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:24,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:24,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:24,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:24,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:24,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:24,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:24,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:56:24,194 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:56:24,194 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:56:24,194 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:56:24,194 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:56:24,194 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:56:24,194 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:56:24,195 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:24,195 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:24,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:24,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:24,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:24,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:24,195 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:24,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:24,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:24,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:24,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:24,196 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:24,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:24,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:24,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:24,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:24,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:24,197 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:24,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:24,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:24,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:24,199 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:24,199 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:24,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:24,199 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:56:24,199 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:56:24,199 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:24,199 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:24,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:24,199 DEBUG [Thread-776] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:56:24,199 DEBUG [Thread-776] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:56:24,199 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:24,199 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:24,199 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:24,199 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:24,199 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:24,199 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:24,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:24,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:24,270 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:24,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55216 deadline: 1679399844267, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:24,271 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:24,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55208 deadline: 1679399844270, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:24,299 DEBUG [Thread-776] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:56:24,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:56:24,301 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:56:24,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:56:24,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:56:24,302 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:56:24,302 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:24,302 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:56:24,302 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:56:24,302 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:24,303 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:24,303 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:24,303 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:24,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:24,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:24,303 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:24,303 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:56:24,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:24,304 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:24,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:56:24,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:24,304 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:56:24,304 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:56:24,304 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:56:24,304 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:56:24,304 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:24,304 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:56:24,304 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:56:24,305 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:56:24,305 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. 2023-03-21 11:56:24,305 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. started... 2023-03-21 11:56:24,305 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. as already flushing 2023-03-21 11:56:24,305 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. 2023-03-21 11:56:24,305 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:56:24,305 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:56:24,305 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:24,305 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:56:24,305 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:56:24,306 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:24,306 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:24,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:24,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:24,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:24,306 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:24,306 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:56:24,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:24,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:24,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:24,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:24,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:24,307 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:56:24,307 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:56:24,307 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b8baac5[Count = 0] remaining members to acquire global barrier 2023-03-21 11:56:24,307 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:24,308 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:24,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:24,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:24,308 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:56:24,308 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:56:24,308 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:56:24,308 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:24,308 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:56:24,309 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:24,309 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:56:24,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:24,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:24,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:24,309 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:24,309 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:56:24,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:24,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:24,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:24,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:24,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:24,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:24,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:24,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:56:24,311 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:56:24,311 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:56:24,311 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:56:24,311 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:56:24,311 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:56:24,311 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:56:24,312 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:24,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:24,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:24,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:24,312 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:24,312 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:24,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:24,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:24,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:24,312 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:24,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:24,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:24,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:24,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:24,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:24,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:24,313 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:24,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:24,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:24,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:24,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:24,315 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:24,315 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:24,315 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:24,315 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:24,315 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:56:24,316 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:56:24,315 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:24,316 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:24,316 DEBUG [Thread-776] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:56:24,316 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:24,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:24,316 DEBUG [Thread-776] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:56:24,316 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:24,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:24,316 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:24,316 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:24,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:24,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:24,316 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:56:24,416 DEBUG [Thread-776] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:56:24,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:56:24,418 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:56:24,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:56:24,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:56:24,419 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:56:24,419 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:24,419 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:56:24,419 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:56:24,420 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:24,420 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:24,420 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:24,420 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:24,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:24,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:24,420 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:24,421 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:56:24,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:24,421 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:24,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:56:24,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:24,421 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:56:24,421 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:56:24,421 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:56:24,421 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:56:24,421 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:24,421 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:56:24,422 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:56:24,422 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:56:24,422 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. 2023-03-21 11:56:24,422 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. started... 2023-03-21 11:56:24,422 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. as already flushing 2023-03-21 11:56:24,422 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. 2023-03-21 11:56:24,422 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:56:24,422 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:56:24,422 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:24,422 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:56:24,422 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:56:24,423 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:24,423 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:24,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:24,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:24,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:24,423 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:24,423 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:56:24,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:24,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:24,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:24,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:24,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:24,432 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:56:24,432 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@38379e8d[Count = 0] remaining members to acquire global barrier 2023-03-21 11:56:24,432 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:56:24,432 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:24,448 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:24,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:24,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:24,448 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:56:24,448 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:56:24,448 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:56:24,448 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:24,448 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:56:24,450 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:56:24,450 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:24,450 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:56:24,450 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:24,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:24,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:24,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:24,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:24,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:24,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:24,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:24,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:24,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:24,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:24,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:56:24,460 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:56:24,460 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:56:24,460 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:56:24,460 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:56:24,460 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:56:24,460 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:56:24,464 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:24,464 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:24,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:24,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:24,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:24,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:24,464 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:24,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:24,465 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:24,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:24,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:24,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:24,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:24,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:24,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:24,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:24,466 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:24,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:24,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:24,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:24,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:24,467 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:24,467 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:24,467 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:24,467 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:24,467 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:24,467 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:24,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:24,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:24,467 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:24,467 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:56:24,468 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:56:24,467 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:24,468 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:24,468 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:24,468 DEBUG [Thread-776] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:56:24,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:24,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:24,468 DEBUG [Thread-776] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:56:24,472 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:24,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55208 deadline: 1679399844472, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:24,472 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:24,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55216 deadline: 1679399844472, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:24,566 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230321fb6b86547591410fa4a56170d7d1cedf to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230321fb6b86547591410fa4a56170d7d1cedf 2023-03-21 11:56:24,568 DEBUG [Thread-776] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:56:24,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:56:24,570 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=366, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/A/c53a5d42d5944973bb18646b587243be 2023-03-21 11:56:24,572 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:56:24,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:56:24,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:56:24,573 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:56:24,573 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:24,573 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:56:24,573 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:56:24,573 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:24,573 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:24,574 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:24,574 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:24,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:24,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:24,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:24,575 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:24,575 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:56:24,575 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:24,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:56:24,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:24,575 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:56:24,575 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:56:24,575 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:56:24,575 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:56:24,575 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:24,576 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:56:24,576 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:56:24,576 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:56:24,576 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. 2023-03-21 11:56:24,576 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. started... 2023-03-21 11:56:24,576 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. as already flushing 2023-03-21 11:56:24,576 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. 2023-03-21 11:56:24,576 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:56:24,576 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:56:24,576 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:24,576 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:56:24,576 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:56:24,577 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:24,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:24,577 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:24,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:24,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:24,577 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:24,577 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:56:24,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:24,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:24,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:24,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:24,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:24,578 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:56:24,578 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@234f067c[Count = 0] remaining members to acquire global barrier 2023-03-21 11:56:24,578 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:56:24,578 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:24,579 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:24,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:24,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:24,579 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:56:24,579 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:56:24,579 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:56:24,579 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:24,579 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:56:24,580 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:56:24,580 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:24,580 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:56:24,580 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:24,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:24,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:24,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:24,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:24,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:24,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:24,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:24,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:24,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:24,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:24,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:56:24,582 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:56:24,582 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:56:24,582 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:56:24,582 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:56:24,582 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:56:24,582 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:56:24,583 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:24,583 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:24,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:24,583 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:24,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:24,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:24,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:24,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:24,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:24,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:24,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:24,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:24,584 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:24,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:24,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:24,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:24,586 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:24,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:24,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:24,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:24,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:24,588 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:24,588 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:24,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:24,588 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:56:24,588 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:56:24,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:24,588 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:24,588 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:24,589 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:24,589 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:24,589 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:24,589 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:24,589 DEBUG [Thread-776] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:56:24,589 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:24,589 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:24,589 DEBUG [Thread-776] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:56:24,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:24,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:24,590 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=366 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/B/2a0a78acb1a04698bc45b5a3f581a184 2023-03-21 11:56:24,600 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=366 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/C/7f6536c556fc44148be999dd99d9c8e2 2023-03-21 11:56:24,605 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/A/c53a5d42d5944973bb18646b587243be as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/c53a5d42d5944973bb18646b587243be 2023-03-21 11:56:24,610 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/c53a5d42d5944973bb18646b587243be, entries=150, sequenceid=366, filesize=25.4 K 2023-03-21 11:56:24,611 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/B/2a0a78acb1a04698bc45b5a3f581a184 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/2a0a78acb1a04698bc45b5a3f581a184 2023-03-21 11:56:24,614 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/2a0a78acb1a04698bc45b5a3f581a184, entries=150, sequenceid=366, filesize=11.9 K 2023-03-21 11:56:24,615 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/C/7f6536c556fc44148be999dd99d9c8e2 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/7f6536c556fc44148be999dd99d9c8e2 2023-03-21 11:56:24,619 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/7f6536c556fc44148be999dd99d9c8e2, entries=150, sequenceid=366, filesize=11.9 K 2023-03-21 11:56:24,620 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for 8041406d0a1ddc5b9409edb79457d352 in 466ms, sequenceid=366, compaction requested=true 2023-03-21 11:56:24,620 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8041406d0a1ddc5b9409edb79457d352: 2023-03-21 11:56:24,620 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-21 11:56:24,620 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-21 11:56:24,620 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-21 11:56:24,620 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-21 11:56:24,620 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-21 11:56:24,621 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 104730 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-21 11:56:24,621 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1912): 8041406d0a1ddc5b9409edb79457d352/A is initiating minor compaction (all files) 2023-03-21 11:56:24,621 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8041406d0a1ddc5b9409edb79457d352/A in TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. 2023-03-21 11:56:24,621 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49290 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-21 11:56:24,622 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/33306f14338d42e5921bbcaefbdaf412, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/f00401a98c504169bc10f4f32bb3979b, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/a0fe0dc5e09b4490b1492cfce690fcd3, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/c53a5d42d5944973bb18646b587243be] into tmpdir=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp, totalSize=102.3 K 2023-03-21 11:56:24,622 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1912): 8041406d0a1ddc5b9409edb79457d352/B is initiating minor compaction (all files) 2023-03-21 11:56:24,622 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8041406d0a1ddc5b9409edb79457d352/B in TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. 2023-03-21 11:56:24,622 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/5bf29df8ddd84de8bc87ca4ce1aa0d71, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/7a8f30c454494ff2b803cb440c4d4180, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/41e3b24dd07644448be2c272436db052, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/2a0a78acb1a04698bc45b5a3f581a184] into tmpdir=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp, totalSize=48.1 K 2023-03-21 11:56:24,622 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 33306f14338d42e5921bbcaefbdaf412, keycount=150, bloomtype=ROW, size=26.0 K, encoding=NONE, compression=NONE, seqNum=287, earliestPutTs=1679399779550 2023-03-21 11:56:24,622 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 5bf29df8ddd84de8bc87ca4ce1aa0d71, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=287, earliestPutTs=1679399779550 2023-03-21 11:56:24,622 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting f00401a98c504169bc10f4f32bb3979b, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=315, earliestPutTs=1679399779631 2023-03-21 11:56:24,623 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 7a8f30c454494ff2b803cb440c4d4180, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=315, earliestPutTs=1679399779631 2023-03-21 11:56:24,623 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting a0fe0dc5e09b4490b1492cfce690fcd3, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=329, earliestPutTs=1679399781789 2023-03-21 11:56:24,623 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 41e3b24dd07644448be2c272436db052, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=329, earliestPutTs=1679399781789 2023-03-21 11:56:24,623 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting c53a5d42d5944973bb18646b587243be, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=366, earliestPutTs=1679399783025 2023-03-21 11:56:24,623 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 2a0a78acb1a04698bc45b5a3f581a184, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=366, earliestPutTs=1679399783025 2023-03-21 11:56:24,630 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] throttle.PressureAwareThroughputController(145): 8041406d0a1ddc5b9409edb79457d352#B#compaction#150 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-21 11:56:24,634 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8041406d0a1ddc5b9409edb79457d352#A#compaction#151 average throughput is 3.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-21 11:56:24,689 DEBUG [Thread-776] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:56:24,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:56:24,691 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:56:24,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:56:24,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:56:24,692 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:56:24,692 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:24,692 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:56:24,692 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:56:24,693 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:24,693 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:24,694 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:24,694 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:24,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:24,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:24,694 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:24,694 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:56:24,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:24,694 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:24,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:56:24,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:24,694 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:56:24,695 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:56:24,695 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:56:24,695 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:56:24,695 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:24,695 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:56:24,695 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:56:24,695 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:56:24,695 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. 2023-03-21 11:56:24,695 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. started... 2023-03-21 11:56:24,695 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8041406d0a1ddc5b9409edb79457d352 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-21 11:56:24,695 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8041406d0a1ddc5b9409edb79457d352, store=A 2023-03-21 11:56:24,696 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:56:24,696 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8041406d0a1ddc5b9409edb79457d352, store=B 2023-03-21 11:56:24,696 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:56:24,696 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8041406d0a1ddc5b9409edb79457d352, store=C 2023-03-21 11:56:24,696 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:56:24,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(9158): Flush requested on 8041406d0a1ddc5b9409edb79457d352 2023-03-21 11:56:24,777 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. as already flushing 2023-03-21 11:56:24,804 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1679399784791, max timestamp=1679399784802Num uniques 859; 2023-03-21 11:56:24,804 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-21 11:56:24,804 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1679399784791, max timestamp=1679399784802Num uniques 859; 2023-03-21 11:56:24,804 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1679399784791, max timestamp=1679399784802Num uniques 859; 2023-03-21 11:56:24,804 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-21 11:56:24,804 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-21 11:56:24,845 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:24,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55216 deadline: 1679399844842, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:24,846 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:24,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55208 deadline: 1679399844842, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:24,953 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:24,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55216 deadline: 1679399844947, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:24,953 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:24,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55208 deadline: 1679399844947, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:25,049 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/B/bffeaedc76b947449daab17099b26e8a as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/bffeaedc76b947449daab17099b26e8a 2023-03-21 11:56:25,055 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 8041406d0a1ddc5b9409edb79457d352/B of 8041406d0a1ddc5b9409edb79457d352 into bffeaedc76b947449daab17099b26e8a(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-21 11:56:25,055 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8041406d0a1ddc5b9409edb79457d352: 2023-03-21 11:56:25,055 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352., storeName=8041406d0a1ddc5b9409edb79457d352/B, priority=12, startTime=1679399784620; duration=0sec 2023-03-21 11:56:25,055 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-21 11:56:25,055 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-21 11:56:25,056 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49290 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-21 11:56:25,056 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1912): 8041406d0a1ddc5b9409edb79457d352/C is initiating minor compaction (all files) 2023-03-21 11:56:25,056 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8041406d0a1ddc5b9409edb79457d352/C in TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. 2023-03-21 11:56:25,056 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/5a814a018ec7427cab20a2fd2e327aa5, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/912fcc93b4ba433e946c283da6ac814c, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/70176395a5da41b2aca0a62c1e7f224b, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/7f6536c556fc44148be999dd99d9c8e2] into tmpdir=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp, totalSize=48.1 K 2023-03-21 11:56:25,057 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 5a814a018ec7427cab20a2fd2e327aa5, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=287, earliestPutTs=1679399779550 2023-03-21 11:56:25,057 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 912fcc93b4ba433e946c283da6ac814c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=315, earliestPutTs=1679399779631 2023-03-21 11:56:25,057 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 70176395a5da41b2aca0a62c1e7f224b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=329, earliestPutTs=1679399781789 2023-03-21 11:56:25,058 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 7f6536c556fc44148be999dd99d9c8e2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=366, earliestPutTs=1679399783025 2023-03-21 11:56:25,065 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] throttle.PressureAwareThroughputController(145): 8041406d0a1ddc5b9409edb79457d352#C#compaction#153 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-21 11:56:25,079 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/C/532ffe8d1afa468b97f963ba7d24c246 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/532ffe8d1afa468b97f963ba7d24c246 2023-03-21 11:56:25,085 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 8041406d0a1ddc5b9409edb79457d352/C of 8041406d0a1ddc5b9409edb79457d352 into 532ffe8d1afa468b97f963ba7d24c246(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-21 11:56:25,085 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8041406d0a1ddc5b9409edb79457d352: 2023-03-21 11:56:25,085 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352., storeName=8041406d0a1ddc5b9409edb79457d352/C, priority=12, startTime=1679399784620; duration=0sec 2023-03-21 11:56:25,085 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-21 11:56:25,133 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230321a6cff828ebb14684b7d0d3182d006578 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230321a6cff828ebb14684b7d0d3182d006578 2023-03-21 11:56:25,157 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:25,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55216 deadline: 1679399845154, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:25,157 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:25,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55208 deadline: 1679399845155, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:25,458 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/A/ce7f828a06c045999146b4d066b968eb as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/ce7f828a06c045999146b4d066b968eb 2023-03-21 11:56:25,462 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 8041406d0a1ddc5b9409edb79457d352/A of 8041406d0a1ddc5b9409edb79457d352 into ce7f828a06c045999146b4d066b968eb(size=26.1 K), total size for store is 26.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-21 11:56:25,462 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8041406d0a1ddc5b9409edb79457d352: 2023-03-21 11:56:25,462 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352., storeName=8041406d0a1ddc5b9409edb79457d352/A, priority=12, startTime=1679399784620; duration=0sec 2023-03-21 11:56:25,462 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-21 11:56:25,463 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:25,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55216 deadline: 1679399845459, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:25,463 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:25,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55208 deadline: 1679399845459, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:25,549 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=376, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/A/9ccb0dd20798483fa862b7e41daa82bc 2023-03-21 11:56:25,564 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=376 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/B/d2a4eba986fb4a4cb7bed4ae26bfd9dc 2023-03-21 11:56:25,576 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=376 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/C/41a9eb38f3224c4c834c7d43454a6746 2023-03-21 11:56:25,580 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/A/9ccb0dd20798483fa862b7e41daa82bc as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/9ccb0dd20798483fa862b7e41daa82bc 2023-03-21 11:56:25,584 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/9ccb0dd20798483fa862b7e41daa82bc, entries=150, sequenceid=376, filesize=25.4 K 2023-03-21 11:56:25,585 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/B/d2a4eba986fb4a4cb7bed4ae26bfd9dc as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/d2a4eba986fb4a4cb7bed4ae26bfd9dc 2023-03-21 11:56:25,589 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/d2a4eba986fb4a4cb7bed4ae26bfd9dc, entries=150, sequenceid=376, filesize=11.9 K 2023-03-21 11:56:25,590 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/C/41a9eb38f3224c4c834c7d43454a6746 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/41a9eb38f3224c4c834c7d43454a6746 2023-03-21 11:56:25,593 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/41a9eb38f3224c4c834c7d43454a6746, entries=150, sequenceid=376, filesize=11.9 K 2023-03-21 11:56:25,594 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=140.89 KB/144270 for 8041406d0a1ddc5b9409edb79457d352 in 899ms, sequenceid=376, compaction requested=false 2023-03-21 11:56:25,594 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8041406d0a1ddc5b9409edb79457d352: 2023-03-21 11:56:25,594 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. 2023-03-21 11:56:25,595 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:56:25,595 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:56:25,595 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:25,595 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:56:25,595 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:56:25,596 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:25,596 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:25,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:25,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:25,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:25,596 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:25,596 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:56:25,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:25,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:25,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:25,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:25,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:25,597 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:56:25,598 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:56:25,598 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6acc01d1[Count = 0] remaining members to acquire global barrier 2023-03-21 11:56:25,598 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:25,598 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:25,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:25,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:25,598 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:56:25,598 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:56:25,598 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:56:25,599 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:25,599 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:56:25,599 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:25,599 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:56:25,599 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:25,599 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:56:25,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:25,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:25,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:25,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:25,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:25,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:25,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:25,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:25,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:25,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:25,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:56:25,601 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:56:25,601 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:56:25,601 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:56:25,601 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:56:25,601 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:56:25,601 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:56:25,602 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:25,602 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:25,602 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:25,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:25,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:25,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:25,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:25,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:25,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:25,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:25,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:25,603 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:25,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:25,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:25,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:25,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:25,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:25,604 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:25,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:25,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:25,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:25,605 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:25,605 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:25,606 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:25,606 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:25,606 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:56:25,606 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:56:25,606 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:56:25,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:25,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:25,606 DEBUG [Thread-776] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:56:25,606 DEBUG [Thread-776] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:56:25,606 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:25,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:25,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:25,606 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:25,606 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:25,606 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:25,606 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:25,606 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:25,706 DEBUG [Thread-776] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:56:25,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:56:25,708 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:56:25,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:56:25,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:56:25,718 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:56:25,718 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:25,718 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:56:25,718 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:56:25,719 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:25,719 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:25,719 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:25,719 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:25,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:25,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:25,720 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:25,720 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:56:25,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:25,720 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:25,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:56:25,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:25,720 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:56:25,720 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:56:25,721 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:56:25,721 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:56:25,721 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:25,721 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:56:25,723 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:56:25,723 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:56:25,723 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. 2023-03-21 11:56:25,723 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. started... 2023-03-21 11:56:25,723 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8041406d0a1ddc5b9409edb79457d352 3/3 column families, dataSize=140.89 KB heapSize=390.84 KB 2023-03-21 11:56:25,724 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8041406d0a1ddc5b9409edb79457d352, store=A 2023-03-21 11:56:25,724 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-21 11:56:25,724 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8041406d0a1ddc5b9409edb79457d352, store=B 2023-03-21 11:56:25,724 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-21 11:56:25,724 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8041406d0a1ddc5b9409edb79457d352, store=C 2023-03-21 11:56:25,724 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-21 11:56:25,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(9158): Flush requested on 8041406d0a1ddc5b9409edb79457d352 2023-03-21 11:56:25,965 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. as already flushing 2023-03-21 11:56:25,976 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:25,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55216 deadline: 1679399845974, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:25,978 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:25,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55208 deadline: 1679399845975, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:26,078 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:26,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55216 deadline: 1679399846077, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:26,080 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:26,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55208 deadline: 1679399846078, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:26,139 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032162c4fc0257364d61aec6aa0a88a29450 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032162c4fc0257364d61aec6aa0a88a29450 2023-03-21 11:56:26,154 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=414, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/A/5238f859610f4af38d7cf333290195b3 2023-03-21 11:56:26,280 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:26,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55216 deadline: 1679399846280, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:26,281 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:26,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55208 deadline: 1679399846281, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:26,572 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=414 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/B/49861eb576524176b74fbd0615d80591 2023-03-21 11:56:26,580 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=414 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/C/bb5c9f8b0dda4b41a4a1244610111ffc 2023-03-21 11:56:26,583 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:26,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55216 deadline: 1679399846581, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:26,584 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/A/5238f859610f4af38d7cf333290195b3 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/5238f859610f4af38d7cf333290195b3 2023-03-21 11:56:26,585 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:26,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55208 deadline: 1679399846582, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:26,588 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/5238f859610f4af38d7cf333290195b3, entries=150, sequenceid=414, filesize=25.4 K 2023-03-21 11:56:26,589 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/B/49861eb576524176b74fbd0615d80591 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/49861eb576524176b74fbd0615d80591 2023-03-21 11:56:26,592 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/49861eb576524176b74fbd0615d80591, entries=150, sequenceid=414, filesize=11.9 K 2023-03-21 11:56:26,594 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/C/bb5c9f8b0dda4b41a4a1244610111ffc as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/bb5c9f8b0dda4b41a4a1244610111ffc 2023-03-21 11:56:26,597 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/bb5c9f8b0dda4b41a4a1244610111ffc, entries=150, sequenceid=414, filesize=11.9 K 2023-03-21 11:56:26,598 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~390.80 KB/400176, currentSize=46.96 KB/48090 for 8041406d0a1ddc5b9409edb79457d352 in 875ms, sequenceid=414, compaction requested=true 2023-03-21 11:56:26,598 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8041406d0a1ddc5b9409edb79457d352: 2023-03-21 11:56:26,598 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. 2023-03-21 11:56:26,598 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:56:26,598 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:56:26,598 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:26,598 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:56:26,598 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:56:26,599 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:26,599 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:26,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:26,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:26,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:26,599 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:26,599 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:56:26,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:26,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:26,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:26,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:26,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:26,601 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:56:26,601 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c8743cf[Count = 0] remaining members to acquire global barrier 2023-03-21 11:56:26,601 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:56:26,601 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:26,601 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:26,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:26,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:26,601 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:56:26,601 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:56:26,601 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:56:26,601 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:26,602 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:56:26,602 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:56:26,602 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:26,602 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:26,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:26,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:26,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:26,602 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:56:26,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:26,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:26,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:26,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:26,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:26,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:26,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:26,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:56:26,604 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:56:26,604 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:56:26,604 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:56:26,604 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:56:26,604 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:56:26,604 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:56:26,605 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:26,605 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:26,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:26,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:26,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:26,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:26,605 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:26,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:26,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:26,605 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:26,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:26,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:26,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:26,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:26,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:26,607 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:26,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:26,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:26,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:26,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:26,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:26,609 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:26,609 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:26,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:26,609 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:56:26,609 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:56:26,609 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:26,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:26,609 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:56:26,609 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:26,609 DEBUG [Thread-776] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:56:26,609 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:26,609 DEBUG [Thread-776] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:56:26,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:26,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:26,609 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:26,609 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:26,609 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:26,609 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:26,609 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:26,709 DEBUG [Thread-776] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:56:26,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:56:26,711 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:56:26,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:56:26,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:56:26,712 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:56:26,712 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:26,712 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:56:26,712 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:56:26,712 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:26,713 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:26,713 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:26,713 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:26,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:26,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:26,713 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:26,713 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:56:26,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:26,714 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:26,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:56:26,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:26,714 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:56:26,714 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:56:26,714 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:56:26,714 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:56:26,714 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:26,714 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:56:26,715 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:56:26,715 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:56:26,715 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. 2023-03-21 11:56:26,715 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. started... 2023-03-21 11:56:26,715 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8041406d0a1ddc5b9409edb79457d352 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-21 11:56:26,715 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8041406d0a1ddc5b9409edb79457d352, store=A 2023-03-21 11:56:26,715 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:56:26,715 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8041406d0a1ddc5b9409edb79457d352, store=B 2023-03-21 11:56:26,715 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:56:26,715 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8041406d0a1ddc5b9409edb79457d352, store=C 2023-03-21 11:56:26,715 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:56:26,729 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032111df5d1503724ab3a6637c2650111c89 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032111df5d1503724ab3a6637c2650111c89 2023-03-21 11:56:27,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(9158): Flush requested on 8041406d0a1ddc5b9409edb79457d352 2023-03-21 11:56:27,085 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. as already flushing 2023-03-21 11:56:27,111 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679399787091, max timestamp=1679399787110Num uniques 859; 2023-03-21 11:56:27,111 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-21 11:56:27,111 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679399787091, max timestamp=1679399787110Num uniques 859; 2023-03-21 11:56:27,111 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-21 11:56:27,112 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679399787091, max timestamp=1679399787110Num uniques 859; 2023-03-21 11:56:27,112 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-21 11:56:27,132 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:27,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55216 deadline: 1679399847130, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:27,132 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:27,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55208 deadline: 1679399847131, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:27,134 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=424, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/A/4ee36cb114134bb19fcf4e205f1c3204 2023-03-21 11:56:27,235 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:27,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55216 deadline: 1679399847233, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:27,235 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:27,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55208 deadline: 1679399847233, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:27,359 DEBUG [Thread-779] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5d07b2f5 to 127.0.0.1:61761 2023-03-21 11:56:27,359 DEBUG [Thread-779] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-21 11:56:27,360 DEBUG [Thread-777] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4af56759 to 127.0.0.1:61761 2023-03-21 11:56:27,360 DEBUG [Thread-777] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-21 11:56:27,363 DEBUG [Thread-781] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5446e9d6 to 127.0.0.1:61761 2023-03-21 11:56:27,363 DEBUG [Thread-781] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-21 11:56:27,365 DEBUG [Thread-783] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x05e8b710 to 127.0.0.1:61761 2023-03-21 11:56:27,365 DEBUG [Thread-783] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-21 11:56:27,439 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:27,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55216 deadline: 1679399847438, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:27,441 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:27,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55208 deadline: 1679399847441, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:27,545 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=424 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/B/bf019f740d9f4c60890c48ffd9f560a1 2023-03-21 11:56:27,741 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:27,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55216 deadline: 1679399847741, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:27,746 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:27,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55208 deadline: 1679399847745, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:27,804 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:27,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55188 deadline: 1679399847804, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:27,805 DEBUG [Thread-770] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8159 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352., hostname=jenkins-hbase20.apache.org,34369,1679399732030, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-21 11:56:27,815 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:27,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55180 deadline: 1679399847814, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:27,816 DEBUG [Thread-768] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8166 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352., hostname=jenkins-hbase20.apache.org,34369,1679399732030, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-21 11:56:27,857 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:27,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55190 deadline: 1679399847857, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:27,859 DEBUG [Thread-772] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8211 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352., hostname=jenkins-hbase20.apache.org,34369,1679399732030, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-21 11:56:27,964 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=424 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/C/7d96d940ff4147c991a719139f7151c8 2023-03-21 11:56:27,975 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/A/4ee36cb114134bb19fcf4e205f1c3204 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/4ee36cb114134bb19fcf4e205f1c3204 2023-03-21 11:56:27,983 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/4ee36cb114134bb19fcf4e205f1c3204, entries=150, sequenceid=424, filesize=25.4 K 2023-03-21 11:56:27,984 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/B/bf019f740d9f4c60890c48ffd9f560a1 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/bf019f740d9f4c60890c48ffd9f560a1 2023-03-21 11:56:27,988 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/bf019f740d9f4c60890c48ffd9f560a1, entries=150, sequenceid=424, filesize=11.9 K 2023-03-21 11:56:27,989 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/C/7d96d940ff4147c991a719139f7151c8 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/7d96d940ff4147c991a719139f7151c8 2023-03-21 11:56:27,995 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/7d96d940ff4147c991a719139f7151c8, entries=150, sequenceid=424, filesize=11.9 K 2023-03-21 11:56:27,996 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=140.89 KB/144270 for 8041406d0a1ddc5b9409edb79457d352 in 1281ms, sequenceid=424, compaction requested=true 2023-03-21 11:56:27,996 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8041406d0a1ddc5b9409edb79457d352: 2023-03-21 11:56:27,996 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. 2023-03-21 11:56:27,996 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:56:27,996 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:56:27,996 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:27,996 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:56:27,996 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:56:27,997 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:27,997 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:27,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:27,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:27,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:27,998 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:27,998 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:56:27,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:27,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:27,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:27,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:27,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:28,000 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:56:28,000 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@78c07924[Count = 0] remaining members to acquire global barrier 2023-03-21 11:56:28,000 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:56:28,000 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:28,001 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:28,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:28,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:28,001 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:56:28,001 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:56:28,001 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:56:28,001 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:28,001 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:56:28,002 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:56:28,002 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:28,002 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:28,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:28,002 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:56:28,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:28,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:28,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:28,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:28,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:28,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:28,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:28,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:28,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:28,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:56:28,006 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:56:28,006 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:56:28,006 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:56:28,006 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:56:28,006 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:56:28,006 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:56:28,007 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:28,007 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:28,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:28,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:28,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:28,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:28,007 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:28,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:28,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:28,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:28,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:28,008 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:28,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:28,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:28,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:28,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:28,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:28,009 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:28,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:28,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:28,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:28,011 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:28,011 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:28,011 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:28,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:28,011 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:56:28,012 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:56:28,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:28,011 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:28,011 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:28,012 DEBUG [Thread-776] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:56:28,012 DEBUG [Thread-776] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:56:28,011 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:28,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:28,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:28,012 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:28,012 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:28,012 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:28,012 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:28,112 DEBUG [Thread-776] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:56:28,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:56:28,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(9158): Flush requested on 8041406d0a1ddc5b9409edb79457d352 2023-03-21 11:56:28,249 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8041406d0a1ddc5b9409edb79457d352 3/3 column families, dataSize=147.60 KB heapSize=415.45 KB 2023-03-21 11:56:28,249 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8041406d0a1ddc5b9409edb79457d352, store=A 2023-03-21 11:56:28,249 DEBUG [Thread-774] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6322d500 to 127.0.0.1:61761 2023-03-21 11:56:28,249 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-21 11:56:28,249 DEBUG [Thread-774] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-21 11:56:28,249 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8041406d0a1ddc5b9409edb79457d352, store=B 2023-03-21 11:56:28,250 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-21 11:56:28,250 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8041406d0a1ddc5b9409edb79457d352, store=C 2023-03-21 11:56:28,250 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-21 11:56:28,253 DEBUG [Thread-766] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x58b82304 to 127.0.0.1:61761 2023-03-21 11:56:28,253 DEBUG [Thread-766] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-21 11:56:28,668 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032181cb86f9de74460db059c85a8f62a284 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032181cb86f9de74460db059c85a8f62a284 2023-03-21 11:56:29,075 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=461, memsize=49.2 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/A/f569b3355eee402e96db6d1c340e5df8 2023-03-21 11:56:29,496 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=461 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/B/ebb752cd6af44e6098ee5c706a05fdfe 2023-03-21 11:56:29,916 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=461 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/C/a65a326a22dd44dd97acbb41ee082b50 2023-03-21 11:56:29,929 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/A/f569b3355eee402e96db6d1c340e5df8 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/f569b3355eee402e96db6d1c340e5df8 2023-03-21 11:56:29,935 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/f569b3355eee402e96db6d1c340e5df8, entries=150, sequenceid=461, filesize=25.4 K 2023-03-21 11:56:29,936 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/B/ebb752cd6af44e6098ee5c706a05fdfe as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/ebb752cd6af44e6098ee5c706a05fdfe 2023-03-21 11:56:29,941 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/ebb752cd6af44e6098ee5c706a05fdfe, entries=150, sequenceid=461, filesize=11.9 K 2023-03-21 11:56:29,942 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/C/a65a326a22dd44dd97acbb41ee082b50 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/a65a326a22dd44dd97acbb41ee082b50 2023-03-21 11:56:29,946 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/a65a326a22dd44dd97acbb41ee082b50, entries=150, sequenceid=461, filesize=11.9 K 2023-03-21 11:56:29,947 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~415.41 KB/425376, currentSize=6.71 KB/6870 for 8041406d0a1ddc5b9409edb79457d352 in 1698ms, sequenceid=461, compaction requested=true 2023-03-21 11:56:29,947 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8041406d0a1ddc5b9409edb79457d352: 2023-03-21 11:56:29,947 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-21 11:56:29,947 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-21 11:56:29,947 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-21 11:56:29,947 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-21 11:56:29,947 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-21 11:56:29,948 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 130912 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-21 11:56:29,948 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1912): 8041406d0a1ddc5b9409edb79457d352/A is initiating minor compaction (all files) 2023-03-21 11:56:29,949 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61612 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-21 11:56:29,949 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8041406d0a1ddc5b9409edb79457d352/A in TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. 2023-03-21 11:56:29,949 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1912): 8041406d0a1ddc5b9409edb79457d352/B is initiating minor compaction (all files) 2023-03-21 11:56:29,949 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8041406d0a1ddc5b9409edb79457d352/B in TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. 2023-03-21 11:56:29,949 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/ce7f828a06c045999146b4d066b968eb, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/9ccb0dd20798483fa862b7e41daa82bc, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/5238f859610f4af38d7cf333290195b3, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/4ee36cb114134bb19fcf4e205f1c3204, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/f569b3355eee402e96db6d1c340e5df8] into tmpdir=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp, totalSize=127.8 K 2023-03-21 11:56:29,949 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/bffeaedc76b947449daab17099b26e8a, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/d2a4eba986fb4a4cb7bed4ae26bfd9dc, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/49861eb576524176b74fbd0615d80591, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/bf019f740d9f4c60890c48ffd9f560a1, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/ebb752cd6af44e6098ee5c706a05fdfe] into tmpdir=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp, totalSize=60.2 K 2023-03-21 11:56:29,949 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting ce7f828a06c045999146b4d066b968eb, keycount=150, bloomtype=ROW, size=26.1 K, encoding=NONE, compression=NONE, seqNum=366, earliestPutTs=1679399783025 2023-03-21 11:56:29,949 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting bffeaedc76b947449daab17099b26e8a, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=366, earliestPutTs=1679399783025 2023-03-21 11:56:29,949 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 9ccb0dd20798483fa862b7e41daa82bc, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=376, earliestPutTs=1679399784156 2023-03-21 11:56:29,949 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting d2a4eba986fb4a4cb7bed4ae26bfd9dc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=376, earliestPutTs=1679399784156 2023-03-21 11:56:29,950 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 5238f859610f4af38d7cf333290195b3, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=414, earliestPutTs=1679399784822 2023-03-21 11:56:29,950 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 49861eb576524176b74fbd0615d80591, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=414, earliestPutTs=1679399784822 2023-03-21 11:56:29,950 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 4ee36cb114134bb19fcf4e205f1c3204, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=424, earliestPutTs=1679399785972 2023-03-21 11:56:29,950 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting bf019f740d9f4c60890c48ffd9f560a1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=424, earliestPutTs=1679399785972 2023-03-21 11:56:29,950 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting f569b3355eee402e96db6d1c340e5df8, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=461, earliestPutTs=1679399787091 2023-03-21 11:56:29,950 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting ebb752cd6af44e6098ee5c706a05fdfe, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=461, earliestPutTs=1679399787091 2023-03-21 11:56:29,960 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] throttle.PressureAwareThroughputController(145): 8041406d0a1ddc5b9409edb79457d352#B#compaction#165 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-21 11:56:29,963 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8041406d0a1ddc5b9409edb79457d352#A#compaction#166 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-21 11:56:30,377 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/B/17e1b081b9df47a485d9b6c01e74bb38 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/17e1b081b9df47a485d9b6c01e74bb38 2023-03-21 11:56:30,383 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 8041406d0a1ddc5b9409edb79457d352/B of 8041406d0a1ddc5b9409edb79457d352 into 17e1b081b9df47a485d9b6c01e74bb38(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-21 11:56:30,384 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8041406d0a1ddc5b9409edb79457d352: 2023-03-21 11:56:30,384 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352., storeName=8041406d0a1ddc5b9409edb79457d352/B, priority=11, startTime=1679399789947; duration=0sec 2023-03-21 11:56:30,384 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-21 11:56:30,384 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-21 11:56:30,386 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61612 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-21 11:56:30,386 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1912): 8041406d0a1ddc5b9409edb79457d352/C is initiating minor compaction (all files) 2023-03-21 11:56:30,386 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8041406d0a1ddc5b9409edb79457d352/C in TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. 2023-03-21 11:56:30,386 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/532ffe8d1afa468b97f963ba7d24c246, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/41a9eb38f3224c4c834c7d43454a6746, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/bb5c9f8b0dda4b41a4a1244610111ffc, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/7d96d940ff4147c991a719139f7151c8, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/a65a326a22dd44dd97acbb41ee082b50] into tmpdir=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp, totalSize=60.2 K 2023-03-21 11:56:30,387 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 532ffe8d1afa468b97f963ba7d24c246, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=366, earliestPutTs=1679399783025 2023-03-21 11:56:30,388 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 41a9eb38f3224c4c834c7d43454a6746, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=376, earliestPutTs=1679399784156 2023-03-21 11:56:30,388 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting bb5c9f8b0dda4b41a4a1244610111ffc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=414, earliestPutTs=1679399784822 2023-03-21 11:56:30,389 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 7d96d940ff4147c991a719139f7151c8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=424, earliestPutTs=1679399785972 2023-03-21 11:56:30,389 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting a65a326a22dd44dd97acbb41ee082b50, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=461, earliestPutTs=1679399787091 2023-03-21 11:56:30,402 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] throttle.PressureAwareThroughputController(145): 8041406d0a1ddc5b9409edb79457d352#C#compaction#167 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-21 11:56:30,816 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/C/7edd594929034987982f9f728f9d32c4 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/7edd594929034987982f9f728f9d32c4 2023-03-21 11:56:30,823 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 8041406d0a1ddc5b9409edb79457d352/C of 8041406d0a1ddc5b9409edb79457d352 into 7edd594929034987982f9f728f9d32c4(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-21 11:56:30,823 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8041406d0a1ddc5b9409edb79457d352: 2023-03-21 11:56:30,823 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352., storeName=8041406d0a1ddc5b9409edb79457d352/C, priority=11, startTime=1679399789947; duration=0sec 2023-03-21 11:56:30,823 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-21 11:56:31,193 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/A/2cc6678c6e0c40f28c8aeb5fb888e837 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/2cc6678c6e0c40f28c8aeb5fb888e837 2023-03-21 11:56:31,197 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 8041406d0a1ddc5b9409edb79457d352/A of 8041406d0a1ddc5b9409edb79457d352 into 2cc6678c6e0c40f28c8aeb5fb888e837(size=26.3 K), total size for store is 26.3 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-03-21 11:56:31,197 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8041406d0a1ddc5b9409edb79457d352: 2023-03-21 11:56:31,197 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352., storeName=8041406d0a1ddc5b9409edb79457d352/A, priority=11, startTime=1679399789947; duration=1sec 2023-03-21 11:56:31,197 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-21 11:56:33,562 DEBUG [regionserver/jenkins-hbase20:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.0, tune throughput to 50.00 MB/second 2023-03-21 11:56:37,839 DEBUG [Thread-768] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6dc16927 to 127.0.0.1:61761 2023-03-21 11:56:37,839 DEBUG [Thread-768] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-21 11:56:37,858 DEBUG [Thread-770] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x031d0ca0 to 127.0.0.1:61761 2023-03-21 11:56:37,858 DEBUG [Thread-770] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-21 11:56:37,936 DEBUG [Thread-772] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x029d68f7 to 127.0.0.1:61761 2023-03-21 11:56:37,937 DEBUG [Thread-772] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-21 11:56:37,937 INFO [Listener at localhost.localdomain/37475] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-03-21 11:56:37,937 INFO [Listener at localhost.localdomain/37475] hbase.AcidGuaranteesTestTool(394): wrote 111 2023-03-21 11:56:37,937 INFO [Listener at localhost.localdomain/37475] hbase.AcidGuaranteesTestTool(394): wrote 50 2023-03-21 11:56:37,937 INFO [Listener at localhost.localdomain/37475] hbase.AcidGuaranteesTestTool(394): wrote 46 2023-03-21 11:56:37,938 INFO [Listener at localhost.localdomain/37475] hbase.AcidGuaranteesTestTool(394): wrote 58 2023-03-21 11:56:37,938 INFO [Listener at localhost.localdomain/37475] hbase.AcidGuaranteesTestTool(394): wrote 120 2023-03-21 11:56:37,938 INFO [Listener at localhost.localdomain/37475] hbase.AcidGuaranteesTestTool(396): Readers: 2023-03-21 11:56:37,938 INFO [Listener at localhost.localdomain/37475] hbase.AcidGuaranteesTestTool(398): read 4334 2023-03-21 11:56:37,940 INFO [Listener at localhost.localdomain/37475] hbase.AcidGuaranteesTestTool(398): read 4255 2023-03-21 11:56:37,940 INFO [Listener at localhost.localdomain/37475] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-03-21 11:56:37,940 INFO [Listener at localhost.localdomain/37475] hbase.AcidGuaranteesTestTool(402): scanned 1831 2023-03-21 11:56:37,940 INFO [Listener at localhost.localdomain/37475] hbase.AcidGuaranteesTestTool(403): verified 5493 rows 2023-03-21 11:56:37,940 INFO [Listener at localhost.localdomain/37475] hbase.AcidGuaranteesTestTool(402): scanned 1843 2023-03-21 11:56:37,941 INFO [Listener at localhost.localdomain/37475] hbase.AcidGuaranteesTestTool(403): verified 5529 rows 2023-03-21 11:56:37,941 INFO [Listener at localhost.localdomain/37475] client.ConnectionImplementation(1959): Closing master protocol: MasterService 2023-03-21 11:56:37,941 DEBUG [Listener at localhost.localdomain/37475] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7dffa3dd to 127.0.0.1:61761 2023-03-21 11:56:37,941 DEBUG [Listener at localhost.localdomain/37475] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-21 11:56:37,943 INFO [Listener at localhost.localdomain/37475] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-03-21 11:56:37,944 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.HMaster$11(2418): Client=jenkins//148.251.75.209 disable TestAcidGuarantees 2023-03-21 11:56:37,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure2.ProcedureExecutor(1029): Stored pid=24, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-03-21 11:56:37,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-03-21 11:56:37,948 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679399797948"}]},"ts":"1679399797948"} 2023-03-21 11:56:37,948 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-03-21 11:56:37,958 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-03-21 11:56:37,958 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=25, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=8041406d0a1ddc5b9409edb79457d352, UNASSIGN}] 2023-03-21 11:56:37,959 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=25, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=8041406d0a1ddc5b9409edb79457d352, UNASSIGN 2023-03-21 11:56:37,960 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=8041406d0a1ddc5b9409edb79457d352, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:37,960 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679399797960"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1679399797960"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679399797960"}]},"ts":"1679399797960"} 2023-03-21 11:56:37,961 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=26, ppid=25, state=RUNNABLE; CloseRegionProcedure 8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030}] 2023-03-21 11:56:38,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-03-21 11:56:38,113 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 8041406d0a1ddc5b9409edb79457d352 2023-03-21 11:56:38,114 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 8041406d0a1ddc5b9409edb79457d352, disabling compactions & flushes 2023-03-21 11:56:38,114 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. 2023-03-21 11:56:38,114 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. 2023-03-21 11:56:38,114 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. after waiting 0 ms 2023-03-21 11:56:38,114 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. 2023-03-21 11:56:38,115 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing 8041406d0a1ddc5b9409edb79457d352 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-03-21 11:56:38,115 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8041406d0a1ddc5b9409edb79457d352, store=A 2023-03-21 11:56:38,115 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:56:38,115 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8041406d0a1ddc5b9409edb79457d352, store=B 2023-03-21 11:56:38,116 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:56:38,116 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8041406d0a1ddc5b9409edb79457d352, store=C 2023-03-21 11:56:38,116 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:56:38,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-03-21 11:56:38,542 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303213756b28443774af9a0e519e4ec6e3247 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303213756b28443774af9a0e519e4ec6e3247 2023-03-21 11:56:38,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-03-21 11:56:38,950 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=471, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/A/d638eb29bb8d4ab1af5983eab6cfa7c6 2023-03-21 11:56:39,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-03-21 11:56:39,362 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=471 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/B/f23dbc1c4c214cfe82db8c63d9acdf3c 2023-03-21 11:56:39,779 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=471 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/C/e8c9761221ea480a9e45d7371d1a0713 2023-03-21 11:56:39,784 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/A/d638eb29bb8d4ab1af5983eab6cfa7c6 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/d638eb29bb8d4ab1af5983eab6cfa7c6 2023-03-21 11:56:39,790 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/d638eb29bb8d4ab1af5983eab6cfa7c6, entries=150, sequenceid=471, filesize=25.4 K 2023-03-21 11:56:39,791 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/B/f23dbc1c4c214cfe82db8c63d9acdf3c as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/f23dbc1c4c214cfe82db8c63d9acdf3c 2023-03-21 11:56:39,797 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/f23dbc1c4c214cfe82db8c63d9acdf3c, entries=150, sequenceid=471, filesize=11.9 K 2023-03-21 11:56:39,798 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/.tmp/C/e8c9761221ea480a9e45d7371d1a0713 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/e8c9761221ea480a9e45d7371d1a0713 2023-03-21 11:56:39,803 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/e8c9761221ea480a9e45d7371d1a0713, entries=150, sequenceid=471, filesize=11.9 K 2023-03-21 11:56:39,803 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for 8041406d0a1ddc5b9409edb79457d352 in 1689ms, sequenceid=471, compaction requested=false 2023-03-21 11:56:39,824 DEBUG [StoreCloser-TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/b1457dfd3fc0408281a28e09e36d2169, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/7c1af716a9f848afa158ef97aea37252, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/27e21010df214d5c8ea61b10d07a0e7c, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/47c811f81290442490e545740b1bd5a1, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/83e40448e5a6453bb520cb8813d3fb16, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/44f66262d69c48b6a05636671dd1f61c, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/ee0fa6f3e3214277b04ba83e1ddebe6a, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/7a7d86f264484f6ba58a5e72773aceeb, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/9ecd9066fc3c4dc4a42e6327f7be7e91, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/273a452d436748db9b85a0a70c1d5ad1, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/ddd9bddd917c4463a3d0cec5fe0b0f26, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/73626e4eaa22461cb38d637952b66fb0, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/db8ceeba926a496eb0d57baecd82fe7f, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/bd20561c9bba4179b307acf694437b6c, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/dc4edfa403fb45b9afa838a455f3b7ef, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/cc333ece122d4659b167b753dde00f69, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/33306f14338d42e5921bbcaefbdaf412, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/f00401a98c504169bc10f4f32bb3979b, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/a0fe0dc5e09b4490b1492cfce690fcd3, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/ce7f828a06c045999146b4d066b968eb, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/c53a5d42d5944973bb18646b587243be, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/9ccb0dd20798483fa862b7e41daa82bc, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/5238f859610f4af38d7cf333290195b3, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/4ee36cb114134bb19fcf4e205f1c3204, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/f569b3355eee402e96db6d1c340e5df8] to archive 2023-03-21 11:56:39,825 DEBUG [StoreCloser-TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-21 11:56:39,827 DEBUG [StoreCloser-TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/b1457dfd3fc0408281a28e09e36d2169 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/b1457dfd3fc0408281a28e09e36d2169 2023-03-21 11:56:39,828 DEBUG [StoreCloser-TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/7c1af716a9f848afa158ef97aea37252 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/7c1af716a9f848afa158ef97aea37252 2023-03-21 11:56:39,829 DEBUG [StoreCloser-TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/27e21010df214d5c8ea61b10d07a0e7c to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/27e21010df214d5c8ea61b10d07a0e7c 2023-03-21 11:56:39,830 DEBUG [StoreCloser-TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/47c811f81290442490e545740b1bd5a1 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/47c811f81290442490e545740b1bd5a1 2023-03-21 11:56:39,830 DEBUG [StoreCloser-TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/83e40448e5a6453bb520cb8813d3fb16 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/83e40448e5a6453bb520cb8813d3fb16 2023-03-21 11:56:39,831 DEBUG [StoreCloser-TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/44f66262d69c48b6a05636671dd1f61c to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/44f66262d69c48b6a05636671dd1f61c 2023-03-21 11:56:39,832 DEBUG [StoreCloser-TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/ee0fa6f3e3214277b04ba83e1ddebe6a to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/ee0fa6f3e3214277b04ba83e1ddebe6a 2023-03-21 11:56:39,833 DEBUG [StoreCloser-TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/7a7d86f264484f6ba58a5e72773aceeb to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/7a7d86f264484f6ba58a5e72773aceeb 2023-03-21 11:56:39,834 DEBUG [StoreCloser-TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/9ecd9066fc3c4dc4a42e6327f7be7e91 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/9ecd9066fc3c4dc4a42e6327f7be7e91 2023-03-21 11:56:39,835 DEBUG [StoreCloser-TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/273a452d436748db9b85a0a70c1d5ad1 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/273a452d436748db9b85a0a70c1d5ad1 2023-03-21 11:56:39,836 DEBUG [StoreCloser-TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/ddd9bddd917c4463a3d0cec5fe0b0f26 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/ddd9bddd917c4463a3d0cec5fe0b0f26 2023-03-21 11:56:39,838 DEBUG [StoreCloser-TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/73626e4eaa22461cb38d637952b66fb0 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/73626e4eaa22461cb38d637952b66fb0 2023-03-21 11:56:39,839 DEBUG [StoreCloser-TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/db8ceeba926a496eb0d57baecd82fe7f to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/db8ceeba926a496eb0d57baecd82fe7f 2023-03-21 11:56:39,840 DEBUG [StoreCloser-TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/bd20561c9bba4179b307acf694437b6c to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/bd20561c9bba4179b307acf694437b6c 2023-03-21 11:56:39,841 DEBUG [StoreCloser-TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/dc4edfa403fb45b9afa838a455f3b7ef to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/dc4edfa403fb45b9afa838a455f3b7ef 2023-03-21 11:56:39,842 DEBUG [StoreCloser-TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/cc333ece122d4659b167b753dde00f69 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/cc333ece122d4659b167b753dde00f69 2023-03-21 11:56:39,843 DEBUG [StoreCloser-TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/33306f14338d42e5921bbcaefbdaf412 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/33306f14338d42e5921bbcaefbdaf412 2023-03-21 11:56:39,843 DEBUG [StoreCloser-TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/f00401a98c504169bc10f4f32bb3979b to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/f00401a98c504169bc10f4f32bb3979b 2023-03-21 11:56:39,844 DEBUG [StoreCloser-TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/a0fe0dc5e09b4490b1492cfce690fcd3 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/a0fe0dc5e09b4490b1492cfce690fcd3 2023-03-21 11:56:39,845 DEBUG [StoreCloser-TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/ce7f828a06c045999146b4d066b968eb to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/ce7f828a06c045999146b4d066b968eb 2023-03-21 11:56:39,846 DEBUG [StoreCloser-TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/c53a5d42d5944973bb18646b587243be to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/c53a5d42d5944973bb18646b587243be 2023-03-21 11:56:39,847 DEBUG [StoreCloser-TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/9ccb0dd20798483fa862b7e41daa82bc to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/9ccb0dd20798483fa862b7e41daa82bc 2023-03-21 11:56:39,848 DEBUG [StoreCloser-TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/5238f859610f4af38d7cf333290195b3 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/5238f859610f4af38d7cf333290195b3 2023-03-21 11:56:39,849 DEBUG [StoreCloser-TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/4ee36cb114134bb19fcf4e205f1c3204 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/4ee36cb114134bb19fcf4e205f1c3204 2023-03-21 11:56:39,850 DEBUG [StoreCloser-TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/f569b3355eee402e96db6d1c340e5df8 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/f569b3355eee402e96db6d1c340e5df8 2023-03-21 11:56:39,853 DEBUG [StoreCloser-TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/40ba77cc4ae04c7495923c06f5df12d9, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/6d4b7ace0bd54798b215a54de28b75cb, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/05d3c84401744612ba8ac5a5563e06cd, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/a47ca18fe7904b6a9d572f9a8bacdaed, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/eb3995f2026d4b9b8790df5a88aa495f, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/72d5721a41824a4a942803ce2aaf07c0, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/f06c058f1c584497bab1483a933d8fac, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/d91baa12a35140209bc866d667b6947f, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/be0be94d0dba4e219c17a9369250d1f2, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/b3ff180225a542d79a46a38259b0cde5, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/dc4187adff83447e9f945ac9ec074ce7, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/b9c74a7141ed44b990ae5731c4af571f, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/d61116124a914c9994b96eedba8e4634, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/2b671d78ad1347eb9bf2906c128f1f8f, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/1acd85607daa4a8b8d7dde1ef1c9e486, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/5bf29df8ddd84de8bc87ca4ce1aa0d71, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/c23741cb63914d0c92ab2b352daf8299, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/7a8f30c454494ff2b803cb440c4d4180, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/41e3b24dd07644448be2c272436db052, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/bffeaedc76b947449daab17099b26e8a, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/2a0a78acb1a04698bc45b5a3f581a184, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/d2a4eba986fb4a4cb7bed4ae26bfd9dc, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/49861eb576524176b74fbd0615d80591, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/bf019f740d9f4c60890c48ffd9f560a1, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/ebb752cd6af44e6098ee5c706a05fdfe] to archive 2023-03-21 11:56:39,853 DEBUG [StoreCloser-TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-21 11:56:39,855 DEBUG [StoreCloser-TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/40ba77cc4ae04c7495923c06f5df12d9 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/40ba77cc4ae04c7495923c06f5df12d9 2023-03-21 11:56:39,857 DEBUG [StoreCloser-TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/6d4b7ace0bd54798b215a54de28b75cb to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/6d4b7ace0bd54798b215a54de28b75cb 2023-03-21 11:56:39,858 DEBUG [StoreCloser-TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/05d3c84401744612ba8ac5a5563e06cd to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/05d3c84401744612ba8ac5a5563e06cd 2023-03-21 11:56:39,859 DEBUG [StoreCloser-TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/a47ca18fe7904b6a9d572f9a8bacdaed to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/a47ca18fe7904b6a9d572f9a8bacdaed 2023-03-21 11:56:39,859 DEBUG [StoreCloser-TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/eb3995f2026d4b9b8790df5a88aa495f to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/eb3995f2026d4b9b8790df5a88aa495f 2023-03-21 11:56:39,860 DEBUG [StoreCloser-TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/72d5721a41824a4a942803ce2aaf07c0 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/72d5721a41824a4a942803ce2aaf07c0 2023-03-21 11:56:39,861 DEBUG [StoreCloser-TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/f06c058f1c584497bab1483a933d8fac to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/f06c058f1c584497bab1483a933d8fac 2023-03-21 11:56:39,862 DEBUG [StoreCloser-TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/d91baa12a35140209bc866d667b6947f to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/d91baa12a35140209bc866d667b6947f 2023-03-21 11:56:39,864 DEBUG [StoreCloser-TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/be0be94d0dba4e219c17a9369250d1f2 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/be0be94d0dba4e219c17a9369250d1f2 2023-03-21 11:56:39,865 DEBUG [StoreCloser-TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/b3ff180225a542d79a46a38259b0cde5 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/b3ff180225a542d79a46a38259b0cde5 2023-03-21 11:56:39,866 DEBUG [StoreCloser-TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/dc4187adff83447e9f945ac9ec074ce7 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/dc4187adff83447e9f945ac9ec074ce7 2023-03-21 11:56:39,867 DEBUG [StoreCloser-TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/b9c74a7141ed44b990ae5731c4af571f to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/b9c74a7141ed44b990ae5731c4af571f 2023-03-21 11:56:39,868 DEBUG [StoreCloser-TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/d61116124a914c9994b96eedba8e4634 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/d61116124a914c9994b96eedba8e4634 2023-03-21 11:56:39,869 DEBUG [StoreCloser-TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/2b671d78ad1347eb9bf2906c128f1f8f to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/2b671d78ad1347eb9bf2906c128f1f8f 2023-03-21 11:56:39,871 DEBUG [StoreCloser-TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/1acd85607daa4a8b8d7dde1ef1c9e486 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/1acd85607daa4a8b8d7dde1ef1c9e486 2023-03-21 11:56:39,872 DEBUG [StoreCloser-TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/5bf29df8ddd84de8bc87ca4ce1aa0d71 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/5bf29df8ddd84de8bc87ca4ce1aa0d71 2023-03-21 11:56:39,873 DEBUG [StoreCloser-TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/c23741cb63914d0c92ab2b352daf8299 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/c23741cb63914d0c92ab2b352daf8299 2023-03-21 11:56:39,874 DEBUG [StoreCloser-TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/7a8f30c454494ff2b803cb440c4d4180 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/7a8f30c454494ff2b803cb440c4d4180 2023-03-21 11:56:39,875 DEBUG [StoreCloser-TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/41e3b24dd07644448be2c272436db052 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/41e3b24dd07644448be2c272436db052 2023-03-21 11:56:39,876 DEBUG [StoreCloser-TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/bffeaedc76b947449daab17099b26e8a to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/bffeaedc76b947449daab17099b26e8a 2023-03-21 11:56:39,877 DEBUG [StoreCloser-TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/2a0a78acb1a04698bc45b5a3f581a184 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/2a0a78acb1a04698bc45b5a3f581a184 2023-03-21 11:56:39,878 DEBUG [StoreCloser-TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/d2a4eba986fb4a4cb7bed4ae26bfd9dc to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/d2a4eba986fb4a4cb7bed4ae26bfd9dc 2023-03-21 11:56:39,879 DEBUG [StoreCloser-TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/49861eb576524176b74fbd0615d80591 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/49861eb576524176b74fbd0615d80591 2023-03-21 11:56:39,880 DEBUG [StoreCloser-TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/bf019f740d9f4c60890c48ffd9f560a1 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/bf019f740d9f4c60890c48ffd9f560a1 2023-03-21 11:56:39,881 DEBUG [StoreCloser-TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/ebb752cd6af44e6098ee5c706a05fdfe to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/ebb752cd6af44e6098ee5c706a05fdfe 2023-03-21 11:56:39,884 DEBUG [StoreCloser-TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/f3cdb56e0bdb4a9bbb57038bfd3f1f8b, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/c144e9fdcfbc498bb26bce99c8df5d7c, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/93e36abfb16c4fd88667a1278a8b2e5a, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/77dc6fa1beac42d4ae2ca72b468e18dc, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/6264dd4f5b884048bdf3868360a8bd7f, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/73027b46d8c74daabaad8863499b715f, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/e5b73ee5f7b24c09954092922f0bf633, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/03e8f67e07bf471995d004b74327ff81, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/f86db8c925ef4b43a1ab01df1e57e2df, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/0efc776393bb4319a23f66765375ee68, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/ca9222ca5e6e4f3785f88820721dc327, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/27c6a44ae8264084af2ab9acb81dc3e7, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/353108bee7814c769e7b79546aafd405, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/e47c5662eca0464e8132fee808566f96, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/9c7c65720183487f9af10bcf5e13f40a, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/5a814a018ec7427cab20a2fd2e327aa5, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/d5f8bdf4efde44c5956a0939d1a1c85b, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/912fcc93b4ba433e946c283da6ac814c, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/70176395a5da41b2aca0a62c1e7f224b, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/532ffe8d1afa468b97f963ba7d24c246, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/7f6536c556fc44148be999dd99d9c8e2, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/41a9eb38f3224c4c834c7d43454a6746, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/bb5c9f8b0dda4b41a4a1244610111ffc, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/7d96d940ff4147c991a719139f7151c8, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/a65a326a22dd44dd97acbb41ee082b50] to archive 2023-03-21 11:56:39,885 DEBUG [StoreCloser-TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-21 11:56:39,886 DEBUG [StoreCloser-TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/f3cdb56e0bdb4a9bbb57038bfd3f1f8b to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/f3cdb56e0bdb4a9bbb57038bfd3f1f8b 2023-03-21 11:56:39,887 DEBUG [StoreCloser-TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/c144e9fdcfbc498bb26bce99c8df5d7c to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/c144e9fdcfbc498bb26bce99c8df5d7c 2023-03-21 11:56:39,888 DEBUG [StoreCloser-TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/93e36abfb16c4fd88667a1278a8b2e5a to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/93e36abfb16c4fd88667a1278a8b2e5a 2023-03-21 11:56:39,889 DEBUG [StoreCloser-TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/77dc6fa1beac42d4ae2ca72b468e18dc to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/77dc6fa1beac42d4ae2ca72b468e18dc 2023-03-21 11:56:39,890 DEBUG [StoreCloser-TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/6264dd4f5b884048bdf3868360a8bd7f to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/6264dd4f5b884048bdf3868360a8bd7f 2023-03-21 11:56:39,891 DEBUG [StoreCloser-TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/73027b46d8c74daabaad8863499b715f to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/73027b46d8c74daabaad8863499b715f 2023-03-21 11:56:39,892 DEBUG [StoreCloser-TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/e5b73ee5f7b24c09954092922f0bf633 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/e5b73ee5f7b24c09954092922f0bf633 2023-03-21 11:56:39,893 DEBUG [StoreCloser-TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/03e8f67e07bf471995d004b74327ff81 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/03e8f67e07bf471995d004b74327ff81 2023-03-21 11:56:39,894 DEBUG [StoreCloser-TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/f86db8c925ef4b43a1ab01df1e57e2df to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/f86db8c925ef4b43a1ab01df1e57e2df 2023-03-21 11:56:39,895 DEBUG [StoreCloser-TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/0efc776393bb4319a23f66765375ee68 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/0efc776393bb4319a23f66765375ee68 2023-03-21 11:56:39,896 DEBUG [StoreCloser-TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/ca9222ca5e6e4f3785f88820721dc327 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/ca9222ca5e6e4f3785f88820721dc327 2023-03-21 11:56:39,897 DEBUG [StoreCloser-TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/27c6a44ae8264084af2ab9acb81dc3e7 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/27c6a44ae8264084af2ab9acb81dc3e7 2023-03-21 11:56:39,898 DEBUG [StoreCloser-TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/353108bee7814c769e7b79546aafd405 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/353108bee7814c769e7b79546aafd405 2023-03-21 11:56:39,899 DEBUG [StoreCloser-TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/e47c5662eca0464e8132fee808566f96 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/e47c5662eca0464e8132fee808566f96 2023-03-21 11:56:39,900 DEBUG [StoreCloser-TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/9c7c65720183487f9af10bcf5e13f40a to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/9c7c65720183487f9af10bcf5e13f40a 2023-03-21 11:56:39,901 DEBUG [StoreCloser-TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/5a814a018ec7427cab20a2fd2e327aa5 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/5a814a018ec7427cab20a2fd2e327aa5 2023-03-21 11:56:39,902 DEBUG [StoreCloser-TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/d5f8bdf4efde44c5956a0939d1a1c85b to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/d5f8bdf4efde44c5956a0939d1a1c85b 2023-03-21 11:56:39,903 DEBUG [StoreCloser-TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/912fcc93b4ba433e946c283da6ac814c to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/912fcc93b4ba433e946c283da6ac814c 2023-03-21 11:56:39,904 DEBUG [StoreCloser-TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/70176395a5da41b2aca0a62c1e7f224b to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/70176395a5da41b2aca0a62c1e7f224b 2023-03-21 11:56:39,905 DEBUG [StoreCloser-TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/532ffe8d1afa468b97f963ba7d24c246 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/532ffe8d1afa468b97f963ba7d24c246 2023-03-21 11:56:39,906 DEBUG [StoreCloser-TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/7f6536c556fc44148be999dd99d9c8e2 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/7f6536c556fc44148be999dd99d9c8e2 2023-03-21 11:56:39,906 DEBUG [StoreCloser-TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/41a9eb38f3224c4c834c7d43454a6746 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/41a9eb38f3224c4c834c7d43454a6746 2023-03-21 11:56:39,907 DEBUG [StoreCloser-TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/bb5c9f8b0dda4b41a4a1244610111ffc to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/bb5c9f8b0dda4b41a4a1244610111ffc 2023-03-21 11:56:39,908 DEBUG [StoreCloser-TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/7d96d940ff4147c991a719139f7151c8 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/7d96d940ff4147c991a719139f7151c8 2023-03-21 11:56:39,909 DEBUG [StoreCloser-TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/a65a326a22dd44dd97acbb41ee082b50 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/a65a326a22dd44dd97acbb41ee082b50 2023-03-21 11:56:39,913 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/recovered.edits/474.seqid, newMaxSeqId=474, maxSeqId=4 2023-03-21 11:56:39,915 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352. 2023-03-21 11:56:39,915 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 8041406d0a1ddc5b9409edb79457d352: 2023-03-21 11:56:39,917 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 8041406d0a1ddc5b9409edb79457d352 2023-03-21 11:56:39,917 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=8041406d0a1ddc5b9409edb79457d352, regionState=CLOSED 2023-03-21 11:56:39,917 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679399799917"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679399799917"}]},"ts":"1679399799917"} 2023-03-21 11:56:39,920 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=26, resume processing ppid=25 2023-03-21 11:56:39,920 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=26, ppid=25, state=SUCCESS; CloseRegionProcedure 8041406d0a1ddc5b9409edb79457d352, server=jenkins-hbase20.apache.org,34369,1679399732030 in 1.9570 sec 2023-03-21 11:56:39,921 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=25, resume processing ppid=24 2023-03-21 11:56:39,921 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=25, ppid=24, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=8041406d0a1ddc5b9409edb79457d352, UNASSIGN in 1.9620 sec 2023-03-21 11:56:39,922 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679399799922"}]},"ts":"1679399799922"} 2023-03-21 11:56:39,923 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-03-21 11:56:39,924 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-03-21 11:56:39,925 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=24, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.9800 sec 2023-03-21 11:56:40,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-03-21 11:56:40,056 INFO [Listener at localhost.localdomain/37475] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 24 completed 2023-03-21 11:56:40,057 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.HMaster$5(2228): Client=jenkins//148.251.75.209 delete TestAcidGuarantees 2023-03-21 11:56:40,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure2.ProcedureExecutor(1029): Stored pid=27, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-03-21 11:56:40,059 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=27, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-21 11:56:40,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-03-21 11:56:40,060 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=27, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-21 11:56:40,064 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/.tmp/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352 2023-03-21 11:56:40,067 DEBUG [HFileArchiver-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/.tmp/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A, FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/.tmp/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B, FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/.tmp/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C, FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/.tmp/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/recovered.edits] 2023-03-21 11:56:40,069 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/.tmp/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/2cc6678c6e0c40f28c8aeb5fb888e837 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/2cc6678c6e0c40f28c8aeb5fb888e837 2023-03-21 11:56:40,070 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/.tmp/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/d638eb29bb8d4ab1af5983eab6cfa7c6 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/A/d638eb29bb8d4ab1af5983eab6cfa7c6 2023-03-21 11:56:40,072 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/.tmp/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/17e1b081b9df47a485d9b6c01e74bb38 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/17e1b081b9df47a485d9b6c01e74bb38 2023-03-21 11:56:40,074 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/.tmp/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/f23dbc1c4c214cfe82db8c63d9acdf3c to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/B/f23dbc1c4c214cfe82db8c63d9acdf3c 2023-03-21 11:56:40,076 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/.tmp/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/7edd594929034987982f9f728f9d32c4 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/7edd594929034987982f9f728f9d32c4 2023-03-21 11:56:40,077 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/.tmp/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/e8c9761221ea480a9e45d7371d1a0713 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/C/e8c9761221ea480a9e45d7371d1a0713 2023-03-21 11:56:40,080 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/.tmp/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/recovered.edits/474.seqid to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352/recovered.edits/474.seqid 2023-03-21 11:56:40,080 DEBUG [HFileArchiver-5] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/.tmp/data/default/TestAcidGuarantees/8041406d0a1ddc5b9409edb79457d352 2023-03-21 11:56:40,080 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-21 11:56:40,081 DEBUG [PEWorker-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-03-21 11:56:40,081 DEBUG [PEWorker-2] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-03-21 11:56:40,085 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032109130aa3f9df4172b2fc8ce6f9dae646 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032109130aa3f9df4172b2fc8ce6f9dae646 2023-03-21 11:56:40,086 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303211063d2f19581495f9e442d2ef2fa8d76 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303211063d2f19581495f9e442d2ef2fa8d76 2023-03-21 11:56:40,087 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032111df5d1503724ab3a6637c2650111c89 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032111df5d1503724ab3a6637c2650111c89 2023-03-21 11:56:40,088 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230321132b6acb18d348eea41c6f007c10ff3f to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230321132b6acb18d348eea41c6f007c10ff3f 2023-03-21 11:56:40,089 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303211c84657477304428bb53560bfd92d6a0 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303211c84657477304428bb53560bfd92d6a0 2023-03-21 11:56:40,091 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230321280b63f80cd44ae3a92093708b6b92a8 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230321280b63f80cd44ae3a92093708b6b92a8 2023-03-21 11:56:40,092 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303213756b28443774af9a0e519e4ec6e3247 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303213756b28443774af9a0e519e4ec6e3247 2023-03-21 11:56:40,093 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032140c1aeb96b1a4403a55fcf5fba44bf72 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032140c1aeb96b1a4403a55fcf5fba44bf72 2023-03-21 11:56:40,094 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032162c4fc0257364d61aec6aa0a88a29450 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032162c4fc0257364d61aec6aa0a88a29450 2023-03-21 11:56:40,094 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032176f7842d065142f3ae36c66bcddeb3ab to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032176f7842d065142f3ae36c66bcddeb3ab 2023-03-21 11:56:40,095 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032181cb86f9de74460db059c85a8f62a284 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032181cb86f9de74460db059c85a8f62a284 2023-03-21 11:56:40,096 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230321a6cff828ebb14684b7d0d3182d006578 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230321a6cff828ebb14684b7d0d3182d006578 2023-03-21 11:56:40,097 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230321a6ef5802d6f547edaae1b691bc018c22 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230321a6ef5802d6f547edaae1b691bc018c22 2023-03-21 11:56:40,098 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230321a81b6a64603a4277a08b011999e6caec to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230321a81b6a64603a4277a08b011999e6caec 2023-03-21 11:56:40,099 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230321ac91c469b2c944f4b211726ae2257203 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230321ac91c469b2c944f4b211726ae2257203 2023-03-21 11:56:40,100 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230321b3f419f2c86e4ae58a8bfef5c3e857ff to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230321b3f419f2c86e4ae58a8bfef5c3e857ff 2023-03-21 11:56:40,100 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230321ba86542d6f6347588f35c93914c07b73 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230321ba86542d6f6347588f35c93914c07b73 2023-03-21 11:56:40,101 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230321bc9aa8eadc1f40c1b12d993a10f5bd58 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230321bc9aa8eadc1f40c1b12d993a10f5bd58 2023-03-21 11:56:40,102 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230321ea4c1f5f3d26474c8846e67989897fde to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230321ea4c1f5f3d26474c8846e67989897fde 2023-03-21 11:56:40,103 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230321eb3a1327f96444e59f5d4491c8fde291 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230321eb3a1327f96444e59f5d4491c8fde291 2023-03-21 11:56:40,104 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230321f9a09e0ad146419093b147728eee35f8 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230321f9a09e0ad146419093b147728eee35f8 2023-03-21 11:56:40,105 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230321fb6b86547591410fa4a56170d7d1cedf to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230321fb6b86547591410fa4a56170d7d1cedf 2023-03-21 11:56:40,105 DEBUG [PEWorker-2] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-03-21 11:56:40,107 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=27, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-21 11:56:40,109 WARN [PEWorker-2] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-03-21 11:56:40,110 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-03-21 11:56:40,112 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=27, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-21 11:56:40,112 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-03-21 11:56:40,112 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1679399800112"}]},"ts":"9223372036854775807"} 2023-03-21 11:56:40,113 INFO [PEWorker-2] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-03-21 11:56:40,113 DEBUG [PEWorker-2] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 8041406d0a1ddc5b9409edb79457d352, NAME => 'TestAcidGuarantees,,1679399764509.8041406d0a1ddc5b9409edb79457d352.', STARTKEY => '', ENDKEY => ''}] 2023-03-21 11:56:40,113 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-03-21 11:56:40,113 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1679399800113"}]},"ts":"9223372036854775807"} 2023-03-21 11:56:40,115 INFO [PEWorker-2] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-03-21 11:56:40,116 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(130): Finished pid=27, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-21 11:56:40,117 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=27, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 59 msec 2023-03-21 11:56:40,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-03-21 11:56:40,161 INFO [Listener at localhost.localdomain/37475] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 27 completed 2023-03-21 11:56:40,173 INFO [Listener at localhost.localdomain/37475] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobMixedAtomicity Thread=224 (was 232), OpenFileDescriptor=517 (was 566), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=307 (was 315), ProcessCount=167 (was 167), AvailableMemoryMB=9073 (was 9570) 2023-03-21 11:56:40,183 INFO [Listener at localhost.localdomain/37475] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testGetAtomicity Thread=224, OpenFileDescriptor=517, MaxFileDescriptor=60000, SystemLoadAverage=307, ProcessCount=167, AvailableMemoryMB=9073 2023-03-21 11:56:40,184 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-21 11:56:40,185 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.HMaster$4(2112): Client=jenkins//148.251.75.209 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-21 11:56:40,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure2.ProcedureExecutor(1029): Stored pid=28, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-03-21 11:56:40,187 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-03-21 11:56:40,187 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(697): Client=jenkins//148.251.75.209 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 28 2023-03-21 11:56:40,188 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-03-21 11:56:40,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-03-21 11:56:40,189 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/.tmp/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b 2023-03-21 11:56:40,190 DEBUG [HFileArchiver-6] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/.tmp/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b empty. 2023-03-21 11:56:40,190 DEBUG [HFileArchiver-6] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/.tmp/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b 2023-03-21 11:56:40,190 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-21 11:56:40,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-03-21 11:56:40,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-03-21 11:56:40,601 DEBUG [PEWorker-5] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-21 11:56:40,604 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 886ee010948b04e81bf5ede3c083ff8b, NAME => 'TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/.tmp 2023-03-21 11:56:40,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-03-21 11:56:41,012 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-21 11:56:41,012 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 886ee010948b04e81bf5ede3c083ff8b, disabling compactions & flushes 2023-03-21 11:56:41,012 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:41,012 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:41,013 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. after waiting 0 ms 2023-03-21 11:56:41,013 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:41,013 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:41,013 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 886ee010948b04e81bf5ede3c083ff8b: 2023-03-21 11:56:41,018 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-03-21 11:56:41,020 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679399801020"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679399801020"}]},"ts":"1679399801020"} 2023-03-21 11:56:41,022 INFO [PEWorker-5] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-03-21 11:56:41,023 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-03-21 11:56:41,023 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679399801023"}]},"ts":"1679399801023"} 2023-03-21 11:56:41,025 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-03-21 11:56:41,027 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=886ee010948b04e81bf5ede3c083ff8b, ASSIGN}] 2023-03-21 11:56:41,029 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=886ee010948b04e81bf5ede3c083ff8b, ASSIGN 2023-03-21 11:56:41,029 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=886ee010948b04e81bf5ede3c083ff8b, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,34369,1679399732030; forceNewPlan=false, retain=false 2023-03-21 11:56:41,181 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=886ee010948b04e81bf5ede3c083ff8b, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:41,181 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679399801181"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1679399801181"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679399801181"}]},"ts":"1679399801181"} 2023-03-21 11:56:41,184 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=30, ppid=29, state=RUNNABLE; OpenRegionProcedure 886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030}] 2023-03-21 11:56:41,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-03-21 11:56:41,349 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:41,349 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 886ee010948b04e81bf5ede3c083ff8b, NAME => 'TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b.', STARTKEY => '', ENDKEY => ''} 2023-03-21 11:56:41,349 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 886ee010948b04e81bf5ede3c083ff8b 2023-03-21 11:56:41,349 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-21 11:56:41,349 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 886ee010948b04e81bf5ede3c083ff8b 2023-03-21 11:56:41,349 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 886ee010948b04e81bf5ede3c083ff8b 2023-03-21 11:56:41,350 INFO [StoreOpener-886ee010948b04e81bf5ede3c083ff8b-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 886ee010948b04e81bf5ede3c083ff8b 2023-03-21 11:56:41,352 DEBUG [StoreOpener-886ee010948b04e81bf5ede3c083ff8b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A 2023-03-21 11:56:41,352 DEBUG [StoreOpener-886ee010948b04e81bf5ede3c083ff8b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A 2023-03-21 11:56:41,352 INFO [StoreOpener-886ee010948b04e81bf5ede3c083ff8b-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-21 11:56:41,352 INFO [StoreOpener-886ee010948b04e81bf5ede3c083ff8b-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 886ee010948b04e81bf5ede3c083ff8b columnFamilyName A 2023-03-21 11:56:41,353 INFO [StoreOpener-886ee010948b04e81bf5ede3c083ff8b-1] regionserver.HStore(310): Store=886ee010948b04e81bf5ede3c083ff8b/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-21 11:56:41,353 INFO [StoreOpener-886ee010948b04e81bf5ede3c083ff8b-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 886ee010948b04e81bf5ede3c083ff8b 2023-03-21 11:56:41,354 DEBUG [StoreOpener-886ee010948b04e81bf5ede3c083ff8b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B 2023-03-21 11:56:41,354 DEBUG [StoreOpener-886ee010948b04e81bf5ede3c083ff8b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B 2023-03-21 11:56:41,354 INFO [StoreOpener-886ee010948b04e81bf5ede3c083ff8b-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-21 11:56:41,354 INFO [StoreOpener-886ee010948b04e81bf5ede3c083ff8b-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 886ee010948b04e81bf5ede3c083ff8b columnFamilyName B 2023-03-21 11:56:41,354 INFO [StoreOpener-886ee010948b04e81bf5ede3c083ff8b-1] regionserver.HStore(310): Store=886ee010948b04e81bf5ede3c083ff8b/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-21 11:56:41,355 INFO [StoreOpener-886ee010948b04e81bf5ede3c083ff8b-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 886ee010948b04e81bf5ede3c083ff8b 2023-03-21 11:56:41,356 DEBUG [StoreOpener-886ee010948b04e81bf5ede3c083ff8b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C 2023-03-21 11:56:41,356 DEBUG [StoreOpener-886ee010948b04e81bf5ede3c083ff8b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C 2023-03-21 11:56:41,356 INFO [StoreOpener-886ee010948b04e81bf5ede3c083ff8b-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-21 11:56:41,356 INFO [StoreOpener-886ee010948b04e81bf5ede3c083ff8b-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 886ee010948b04e81bf5ede3c083ff8b columnFamilyName C 2023-03-21 11:56:41,356 INFO [StoreOpener-886ee010948b04e81bf5ede3c083ff8b-1] regionserver.HStore(310): Store=886ee010948b04e81bf5ede3c083ff8b/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-21 11:56:41,356 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:41,357 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b 2023-03-21 11:56:41,357 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b 2023-03-21 11:56:41,359 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-21 11:56:41,359 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 886ee010948b04e81bf5ede3c083ff8b 2023-03-21 11:56:41,361 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-21 11:56:41,361 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 886ee010948b04e81bf5ede3c083ff8b; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74720455, jitterRate=0.11342154443264008}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-21 11:56:41,361 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 886ee010948b04e81bf5ede3c083ff8b: 2023-03-21 11:56:41,361 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b., pid=30, masterSystemTime=1679399801337 2023-03-21 11:56:41,363 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:41,363 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:41,363 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=886ee010948b04e81bf5ede3c083ff8b, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:41,363 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679399801363"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1679399801363"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1679399801363"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1679399801363"}]},"ts":"1679399801363"} 2023-03-21 11:56:41,366 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=30, resume processing ppid=29 2023-03-21 11:56:41,366 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=30, ppid=29, state=SUCCESS; OpenRegionProcedure 886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 in 181 msec 2023-03-21 11:56:41,368 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=29, resume processing ppid=28 2023-03-21 11:56:41,368 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=29, ppid=28, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=886ee010948b04e81bf5ede3c083ff8b, ASSIGN in 339 msec 2023-03-21 11:56:41,368 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-03-21 11:56:41,369 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679399801368"}]},"ts":"1679399801368"} 2023-03-21 11:56:41,370 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-03-21 11:56:41,371 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-03-21 11:56:41,372 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=28, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1860 sec 2023-03-21 11:56:42,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-03-21 11:56:42,298 INFO [Listener at localhost.localdomain/37475] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 28 completed 2023-03-21 11:56:42,299 DEBUG [Listener at localhost.localdomain/37475] zookeeper.ReadOnlyZKClient(139): Connect 0x05fbdc11 to 127.0.0.1:61761 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-21 11:56:42,308 DEBUG [Listener at localhost.localdomain/37475] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7ebff1c8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-21 11:56:42,309 DEBUG [Listener at localhost.localdomain/37475] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-21 11:56:42,311 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:41714, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-21 11:56:42,312 DEBUG [Listener at localhost.localdomain/37475] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-03-21 11:56:42,314 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:57020, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-03-21 11:56:42,316 DEBUG [Listener at localhost.localdomain/37475] zookeeper.ReadOnlyZKClient(139): Connect 0x6298fe55 to 127.0.0.1:61761 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-21 11:56:42,331 DEBUG [Listener at localhost.localdomain/37475] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3958bd43, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-21 11:56:42,332 DEBUG [Listener at localhost.localdomain/37475] zookeeper.ReadOnlyZKClient(139): Connect 0x5620345b to 127.0.0.1:61761 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-21 11:56:42,341 DEBUG [Listener at localhost.localdomain/37475] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@73ef029c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-21 11:56:42,342 DEBUG [Listener at localhost.localdomain/37475] zookeeper.ReadOnlyZKClient(139): Connect 0x76e08050 to 127.0.0.1:61761 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-21 11:56:42,348 DEBUG [Listener at localhost.localdomain/37475] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2135e03b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-21 11:56:42,349 DEBUG [Listener at localhost.localdomain/37475] zookeeper.ReadOnlyZKClient(139): Connect 0x2dbba21f to 127.0.0.1:61761 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-21 11:56:42,360 DEBUG [Listener at localhost.localdomain/37475] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5f2b45f8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-21 11:56:42,360 DEBUG [Listener at localhost.localdomain/37475] zookeeper.ReadOnlyZKClient(139): Connect 0x54ba13aa to 127.0.0.1:61761 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-21 11:56:42,367 DEBUG [Listener at localhost.localdomain/37475] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@24e1e878, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-21 11:56:42,368 DEBUG [Listener at localhost.localdomain/37475] zookeeper.ReadOnlyZKClient(139): Connect 0x7e9921a7 to 127.0.0.1:61761 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-21 11:56:42,376 DEBUG [Listener at localhost.localdomain/37475] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5590a424, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-21 11:56:42,376 DEBUG [Listener at localhost.localdomain/37475] zookeeper.ReadOnlyZKClient(139): Connect 0x5ffcf146 to 127.0.0.1:61761 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-21 11:56:42,382 DEBUG [Listener at localhost.localdomain/37475] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@68df7707, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-21 11:56:42,382 DEBUG [Listener at localhost.localdomain/37475] zookeeper.ReadOnlyZKClient(139): Connect 0x3fba9cd6 to 127.0.0.1:61761 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-21 11:56:42,386 DEBUG [Listener at localhost.localdomain/37475] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7742ca6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-21 11:56:42,386 DEBUG [Listener at localhost.localdomain/37475] zookeeper.ReadOnlyZKClient(139): Connect 0x5e897a83 to 127.0.0.1:61761 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-21 11:56:42,389 DEBUG [Listener at localhost.localdomain/37475] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@353e11a8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-21 11:56:42,390 DEBUG [Listener at localhost.localdomain/37475] zookeeper.ReadOnlyZKClient(139): Connect 0x073dacc0 to 127.0.0.1:61761 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-21 11:56:42,394 DEBUG [Listener at localhost.localdomain/37475] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@12f583c6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-21 11:56:42,401 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:56:42,404 DEBUG [hconnection-0x42fd44cf-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-21 11:56:42,405 DEBUG [hconnection-0x1b1cb91d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-21 11:56:42,406 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:41730, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-21 11:56:42,407 DEBUG [hconnection-0x3f854ea1-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-21 11:56:42,407 DEBUG [hconnection-0x43ad8e70-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-21 11:56:42,407 DEBUG [hconnection-0x4aa946f1-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-21 11:56:42,407 DEBUG [hconnection-0x48184a5-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-21 11:56:42,408 DEBUG [hconnection-0x68d71c5a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-21 11:56:42,410 DEBUG [hconnection-0x4ffd2a3c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-21 11:56:42,410 DEBUG [hconnection-0x17d55cb9-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-21 11:56:42,410 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:41746, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-21 11:56:42,411 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:41740, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-21 11:56:42,412 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:41752, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-21 11:56:42,410 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:41744, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-21 11:56:42,411 DEBUG [hconnection-0x4c93fd56-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-21 11:56:42,412 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:41768, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-21 11:56:42,416 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:41798, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-21 11:56:42,416 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:41796, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-21 11:56:42,418 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:41780, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-21 11:56:42,418 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:41826, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-21 11:56:42,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:56:42,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:56:42,420 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:56:42,420 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:42,420 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:56:42,420 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:56:42,423 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:42,423 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:42,424 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:42,424 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:42,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:42,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:42,424 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:42,424 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:56:42,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:42,427 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:42,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:56:42,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:42,427 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:56:42,438 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:56:42,439 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:56:42,439 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:56:42,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(9158): Flush requested on 886ee010948b04e81bf5ede3c083ff8b 2023-03-21 11:56:42,439 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:42,439 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:56:42,439 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:56:42,439 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:56:42,439 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:42,439 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. started... 2023-03-21 11:56:42,439 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. as already flushing 2023-03-21 11:56:42,439 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:42,440 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:56:42,440 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:56:42,440 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:42,440 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:56:42,440 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:56:42,439 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 886ee010948b04e81bf5ede3c083ff8b 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-21 11:56:42,441 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:42,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:42,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:42,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:42,441 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:42,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:42,441 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:42,441 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:56:42,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:42,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:42,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:42,442 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 886ee010948b04e81bf5ede3c083ff8b, store=A 2023-03-21 11:56:42,442 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:56:42,442 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 886ee010948b04e81bf5ede3c083ff8b, store=B 2023-03-21 11:56:42,442 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:56:42,442 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 886ee010948b04e81bf5ede3c083ff8b, store=C 2023-03-21 11:56:42,442 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:56:42,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:42,444 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:56:42,444 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2367fca6[Count = 0] remaining members to acquire global barrier 2023-03-21 11:56:42,444 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:56:42,444 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:42,445 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:42,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:42,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:42,445 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:56:42,445 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:56:42,445 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:56:42,445 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:42,445 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:56:42,446 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:56:42,446 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:42,446 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:56:42,446 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:42,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:42,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:42,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:42,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:42,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:42,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:42,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:42,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:42,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:42,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:42,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:56:42,450 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:56:42,450 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:56:42,450 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:56:42,450 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:56:42,450 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:56:42,450 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:56:42,452 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:42,452 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:42,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:42,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:42,452 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:42,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:42,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:42,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:42,452 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:42,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:42,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:42,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:42,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:42,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:42,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:42,453 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:42,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:42,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:42,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:42,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:42,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:42,455 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:42,455 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:42,455 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:56:42,455 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:56:42,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:42,455 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:42,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:42,455 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:42,455 DEBUG [Thread-1446] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:56:42,455 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:42,455 DEBUG [Thread-1446] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:56:42,455 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:56:42,455 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:42,456 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:42,456 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:42,456 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:42,456 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:42,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:42,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:42,456 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=15 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/A/e563dcb887eb411080fb5ce59692f5ff 2023-03-21 11:56:42,472 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=15 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/B/56296638322246d2b38dc30711b72646 2023-03-21 11:56:42,472 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:42,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41780 deadline: 1679399862470, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:42,474 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:42,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41768 deadline: 1679399862471, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:42,474 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:42,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41826 deadline: 1679399862471, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:42,474 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:42,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41730 deadline: 1679399862472, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:42,475 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:42,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41744 deadline: 1679399862474, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:42,500 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=15 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/C/c7b344cea8784e15b7d70f21e807f05a 2023-03-21 11:56:42,505 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/A/e563dcb887eb411080fb5ce59692f5ff as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/e563dcb887eb411080fb5ce59692f5ff 2023-03-21 11:56:42,510 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/e563dcb887eb411080fb5ce59692f5ff, entries=150, sequenceid=15, filesize=11.6 K 2023-03-21 11:56:42,511 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/B/56296638322246d2b38dc30711b72646 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/56296638322246d2b38dc30711b72646 2023-03-21 11:56:42,514 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/56296638322246d2b38dc30711b72646, entries=150, sequenceid=15, filesize=11.6 K 2023-03-21 11:56:42,515 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/C/c7b344cea8784e15b7d70f21e807f05a as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/c7b344cea8784e15b7d70f21e807f05a 2023-03-21 11:56:42,518 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/c7b344cea8784e15b7d70f21e807f05a, entries=150, sequenceid=15, filesize=11.6 K 2023-03-21 11:56:42,519 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=107.34 KB/109920 for 886ee010948b04e81bf5ede3c083ff8b in 80ms, sequenceid=15, compaction requested=false 2023-03-21 11:56:42,519 DEBUG [MemStoreFlusher.0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-03-21 11:56:42,520 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 886ee010948b04e81bf5ede3c083ff8b: 2023-03-21 11:56:42,556 DEBUG [Thread-1446] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:56:42,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:56:42,557 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:56:42,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:56:42,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:56:42,559 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:56:42,559 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:42,559 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:56:42,559 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:56:42,559 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:42,559 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:42,561 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:42,561 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:42,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:42,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:42,561 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:42,561 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:56:42,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:42,562 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:42,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:56:42,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:42,562 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:56:42,562 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:56:42,562 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:56:42,562 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:56:42,562 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:42,562 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:56:42,570 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:56:42,570 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:56:42,572 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:42,574 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. started... 2023-03-21 11:56:42,574 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 886ee010948b04e81bf5ede3c083ff8b 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-21 11:56:42,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(9158): Flush requested on 886ee010948b04e81bf5ede3c083ff8b 2023-03-21 11:56:42,574 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. as already flushing 2023-03-21 11:56:42,575 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 886ee010948b04e81bf5ede3c083ff8b, store=A 2023-03-21 11:56:42,575 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:56:42,575 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 886ee010948b04e81bf5ede3c083ff8b, store=B 2023-03-21 11:56:42,575 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:56:42,575 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 886ee010948b04e81bf5ede3c083ff8b, store=C 2023-03-21 11:56:42,575 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:56:42,586 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:42,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41744 deadline: 1679399862583, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:42,589 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:42,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41768 deadline: 1679399862585, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:42,589 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:42,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41730 deadline: 1679399862586, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:42,590 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:42,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41780 deadline: 1679399862586, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:42,590 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:42,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41826 deadline: 1679399862586, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:42,687 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:42,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41744 deadline: 1679399862687, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:42,691 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:42,691 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:42,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41768 deadline: 1679399862690, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:42,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41730 deadline: 1679399862690, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:42,692 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:42,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41780 deadline: 1679399862690, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:42,692 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:42,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41826 deadline: 1679399862691, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:42,891 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:42,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41744 deadline: 1679399862889, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:42,894 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:42,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41780 deadline: 1679399862892, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:42,895 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:42,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41768 deadline: 1679399862893, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:42,895 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:42,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41730 deadline: 1679399862893, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:42,895 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:42,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41826 deadline: 1679399862893, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,004 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=35 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/A/69ddd8fc4c6c45078ad26db25a236568 2023-03-21 11:56:43,034 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=35 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/B/32882af9b9af404ea0e29eba90b740f5 2023-03-21 11:56:43,052 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=35 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/C/ff9c1d504a974bb3a8aba60a10dbc165 2023-03-21 11:56:43,057 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/A/69ddd8fc4c6c45078ad26db25a236568 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/69ddd8fc4c6c45078ad26db25a236568 2023-03-21 11:56:43,066 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/69ddd8fc4c6c45078ad26db25a236568, entries=150, sequenceid=35, filesize=11.6 K 2023-03-21 11:56:43,072 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/B/32882af9b9af404ea0e29eba90b740f5 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/32882af9b9af404ea0e29eba90b740f5 2023-03-21 11:56:43,077 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/32882af9b9af404ea0e29eba90b740f5, entries=150, sequenceid=35, filesize=11.6 K 2023-03-21 11:56:43,078 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/C/ff9c1d504a974bb3a8aba60a10dbc165 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/ff9c1d504a974bb3a8aba60a10dbc165 2023-03-21 11:56:43,084 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/ff9c1d504a974bb3a8aba60a10dbc165, entries=150, sequenceid=35, filesize=11.6 K 2023-03-21 11:56:43,085 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 886ee010948b04e81bf5ede3c083ff8b in 510ms, sequenceid=35, compaction requested=false 2023-03-21 11:56:43,086 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 886ee010948b04e81bf5ede3c083ff8b: 2023-03-21 11:56:43,086 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:43,086 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:56:43,087 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:56:43,087 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,087 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:56:43,087 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:56:43,088 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:43,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:43,088 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:43,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:43,089 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:43,089 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:56:43,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:43,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:43,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:43,091 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:56:43,091 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@292fb392[Count = 0] remaining members to acquire global barrier 2023-03-21 11:56:43,091 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:56:43,091 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:43,092 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:43,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:43,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:43,092 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:56:43,092 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:56:43,092 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:56:43,094 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:56:43,094 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:43,094 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:56:43,094 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,094 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:56:43,094 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:43,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:43,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:43,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:43,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:43,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:43,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:43,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:56:43,099 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:56:43,099 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:56:43,099 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:56:43,099 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:56:43,099 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:56:43,099 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:56:43,107 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:43,107 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:43,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:43,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:43,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:43,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:43,109 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:43,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:43,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:43,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:43,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:43,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:43,110 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:43,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:43,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:43,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,112 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:43,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:43,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,116 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,116 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:43,116 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:43,116 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,116 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:43,116 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:43,116 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:43,116 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:43,116 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:43,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:43,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:43,116 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:43,116 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:56:43,117 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:56:43,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:43,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:43,117 DEBUG [Thread-1446] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:56:43,117 DEBUG [Thread-1446] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:56:43,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(9158): Flush requested on 886ee010948b04e81bf5ede3c083ff8b 2023-03-21 11:56:43,196 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 886ee010948b04e81bf5ede3c083ff8b 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-21 11:56:43,196 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 886ee010948b04e81bf5ede3c083ff8b, store=A 2023-03-21 11:56:43,196 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:56:43,196 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 886ee010948b04e81bf5ede3c083ff8b, store=B 2023-03-21 11:56:43,196 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:56:43,196 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 886ee010948b04e81bf5ede3c083ff8b, store=C 2023-03-21 11:56:43,196 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:56:43,214 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:43,214 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:43,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41768 deadline: 1679399863210, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41826 deadline: 1679399863210, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,215 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:43,215 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:43,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41780 deadline: 1679399863211, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41744 deadline: 1679399863211, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,215 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:43,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41730 deadline: 1679399863212, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,217 DEBUG [Thread-1446] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:56:43,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:56:43,219 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:56:43,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:56:43,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:56:43,220 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:56:43,220 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:43,220 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:56:43,220 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:56:43,220 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:43,220 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:43,221 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,221 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:43,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:43,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:43,221 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,221 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:56:43,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:43,221 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:43,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:56:43,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:43,221 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:56:43,222 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:56:43,222 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:56:43,222 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:56:43,222 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:43,222 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:56:43,222 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:56:43,222 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:56:43,222 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:43,222 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. started... 2023-03-21 11:56:43,222 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. as already flushing 2023-03-21 11:56:43,223 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:43,223 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:56:43,223 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:56:43,223 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,223 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:56:43,223 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:56:43,224 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,224 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:43,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:43,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:43,224 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:43,224 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:56:43,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:43,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:43,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:43,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:43,225 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:56:43,225 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ff00443[Count = 0] remaining members to acquire global barrier 2023-03-21 11:56:43,225 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:56:43,225 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:43,225 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:43,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:43,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:43,225 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:56:43,225 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:56:43,225 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,225 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:56:43,225 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:56:43,226 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,226 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:56:43,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:43,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:43,226 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:43,226 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:56:43,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:43,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:43,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:43,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:43,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:43,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:56:43,228 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:56:43,228 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:56:43,228 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:56:43,228 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:56:43,228 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:56:43,228 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:56:43,228 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:43,229 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:43,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:43,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:43,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:43,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:43,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:43,229 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:43,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:43,229 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:43,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:43,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:43,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:43,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:43,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:43,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,230 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:43,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:43,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,232 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,232 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:43,232 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:43,232 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:43,232 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:56:43,232 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:43,232 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:43,232 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:43,232 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:56:43,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:43,232 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,232 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:43,232 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:43,232 DEBUG [Thread-1446] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:56:43,232 DEBUG [Thread-1446] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:56:43,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:43,232 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:43,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:43,317 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:43,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41826 deadline: 1679399863315, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,317 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:43,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41768 deadline: 1679399863315, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,317 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:43,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41780 deadline: 1679399863316, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,317 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:43,317 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:43,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41744 deadline: 1679399863316, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41730 deadline: 1679399863316, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,333 DEBUG [Thread-1446] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:56:43,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:56:43,334 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:56:43,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:56:43,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:56:43,335 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:56:43,335 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:43,335 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:56:43,335 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:56:43,336 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:43,336 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:43,336 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,336 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:43,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:43,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:43,337 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,337 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:56:43,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:43,337 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:43,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:56:43,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:43,337 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:56:43,338 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:56:43,338 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:56:43,338 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:56:43,338 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:43,338 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:56:43,338 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:56:43,338 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:56:43,338 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:43,338 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. started... 2023-03-21 11:56:43,338 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. as already flushing 2023-03-21 11:56:43,338 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:43,338 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:56:43,338 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:56:43,338 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,338 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:56:43,338 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:56:43,340 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:43,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:43,341 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:43,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:43,341 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:43,341 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:56:43,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:43,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:43,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:43,342 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:56:43,342 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:56:43,342 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@140249c0[Count = 0] remaining members to acquire global barrier 2023-03-21 11:56:43,342 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:43,343 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:43,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:43,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:43,343 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,343 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:56:43,343 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:56:43,344 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:56:43,344 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:56:43,344 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,344 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:56:43,344 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:43,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:43,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:43,344 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:56:43,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:43,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:43,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:43,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:43,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:43,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:56:43,346 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:56:43,346 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:56:43,346 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:56:43,346 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:56:43,346 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:56:43,346 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:56:43,347 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:43,347 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:43,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:43,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:43,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:43,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:43,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:43,347 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:43,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:43,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:43,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:43,348 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:43,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:43,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:43,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:43,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,349 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:43,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:43,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,350 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,350 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:43,350 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:43,350 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:43,350 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:56:43,350 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:56:43,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:43,350 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:43,350 DEBUG [Thread-1446] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:56:43,350 DEBUG [Thread-1446] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:56:43,350 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:43,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:43,350 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,351 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:43,351 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:43,351 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:43,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:43,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:43,450 DEBUG [Thread-1446] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:56:43,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:56:43,452 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:56:43,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:56:43,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:56:43,453 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:56:43,453 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:43,453 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:56:43,453 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:56:43,453 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:43,454 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:43,454 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:43,454 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:43,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:43,454 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,454 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:56:43,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:43,455 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:43,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:56:43,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:43,455 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:56:43,455 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:56:43,455 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:56:43,455 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:56:43,455 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:43,455 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:56:43,456 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:56:43,456 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:56:43,456 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:43,456 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. started... 2023-03-21 11:56:43,456 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. as already flushing 2023-03-21 11:56:43,456 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:43,456 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:56:43,456 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:56:43,456 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,456 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:56:43,456 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:56:43,458 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,459 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:43,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:43,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:43,459 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:43,459 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:56:43,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:43,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:43,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:43,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:43,460 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:56:43,460 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@89c052a[Count = 0] remaining members to acquire global barrier 2023-03-21 11:56:43,460 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:56:43,460 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:43,461 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:43,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:43,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:43,461 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:56:43,461 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:56:43,461 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:56:43,461 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,461 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:56:43,461 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,461 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:56:43,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:43,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:43,462 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:43,462 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:56:43,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:43,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:43,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:43,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:43,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:43,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:56:43,463 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:56:43,463 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:56:43,463 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:56:43,463 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:56:43,463 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:56:43,463 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:56:43,464 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:43,464 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:43,464 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:43,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:43,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:43,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:43,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:43,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:43,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:43,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:43,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:43,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:43,465 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:43,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:43,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:43,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,466 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:43,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:43,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,467 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,467 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:43,467 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:43,467 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:43,467 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,467 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:43,467 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:43,467 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:43,467 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:43,467 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:43,468 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:56:43,468 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:56:43,468 DEBUG [Thread-1446] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:56:43,468 DEBUG [Thread-1446] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:56:43,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:43,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:43,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:43,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:43,520 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:43,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41826 deadline: 1679399863518, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,520 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:43,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41768 deadline: 1679399863518, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,520 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:43,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41780 deadline: 1679399863518, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,520 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:43,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41744 deadline: 1679399863519, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,521 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:43,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41730 deadline: 1679399863519, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,568 DEBUG [Thread-1446] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:56:43,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:56:43,570 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:56:43,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:56:43,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:56:43,571 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:56:43,571 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:43,571 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:56:43,571 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:56:43,572 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:43,572 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:43,572 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:43,572 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:43,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:43,573 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,573 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:56:43,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:43,573 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:43,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:56:43,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:43,573 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:56:43,573 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:56:43,574 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:56:43,574 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:56:43,574 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:43,574 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:56:43,574 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:56:43,574 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:56:43,574 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:43,574 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. started... 2023-03-21 11:56:43,574 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. as already flushing 2023-03-21 11:56:43,574 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:43,574 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:56:43,574 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:56:43,574 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,574 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:56:43,574 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:56:43,575 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:43,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:43,575 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:43,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:43,575 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:43,575 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:56:43,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:43,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:43,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:43,576 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:56:43,576 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5cfb9488[Count = 0] remaining members to acquire global barrier 2023-03-21 11:56:43,576 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:56:43,576 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:43,577 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:43,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:43,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:43,577 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:56:43,577 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:56:43,577 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,577 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:56:43,577 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:56:43,578 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,578 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:56:43,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:43,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:43,578 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:43,578 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:56:43,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:43,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:43,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:43,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:43,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:43,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:56:43,580 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:56:43,580 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:56:43,580 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:56:43,580 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:56:43,580 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:56:43,580 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:56:43,581 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:43,581 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:43,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:43,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:43,581 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:43,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:43,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:43,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:43,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:43,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:43,582 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:43,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:43,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:43,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:43,583 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:43,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:43,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:43,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,584 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:43,584 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:43,584 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:43,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:43,584 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:56:43,584 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:56:43,584 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:43,584 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:43,585 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:56:43,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:43,584 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:43,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:43,585 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,585 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:43,585 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:43,585 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:43,585 DEBUG [Thread-1446] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:56:43,585 DEBUG [Thread-1446] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:56:43,605 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=48 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/A/a89eced45b54415a8c79345f21b0521f 2023-03-21 11:56:43,615 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=48 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/B/68df7fa6a7ce4bb48904c30c851a31c1 2023-03-21 11:56:43,685 DEBUG [Thread-1446] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:56:43,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:56:43,687 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:56:43,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:56:43,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:56:43,688 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:56:43,688 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:43,688 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:56:43,688 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:56:43,688 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:43,688 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:43,689 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,689 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:43,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:43,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:43,689 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,689 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:56:43,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:43,689 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:43,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:56:43,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:43,690 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:56:43,690 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:56:43,690 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:56:43,690 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:56:43,690 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:43,690 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:56:43,690 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:56:43,690 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:56:43,690 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:43,690 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. started... 2023-03-21 11:56:43,691 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. as already flushing 2023-03-21 11:56:43,691 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:43,691 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:56:43,691 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:56:43,691 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,691 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:56:43,691 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:56:43,693 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,693 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:43,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:43,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:43,693 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:43,693 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:56:43,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:43,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:43,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:43,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:43,694 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:56:43,695 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@33464148[Count = 0] remaining members to acquire global barrier 2023-03-21 11:56:43,695 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:56:43,695 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:43,695 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:43,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:43,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:43,695 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:56:43,695 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:56:43,696 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:56:43,696 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,696 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:56:43,696 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,696 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:56:43,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:43,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:43,696 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:43,697 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:56:43,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:43,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:43,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:43,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:43,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:43,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:56:43,698 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:56:43,698 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:56:43,698 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:56:43,698 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:56:43,698 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:56:43,698 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:56:43,699 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:43,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:43,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:43,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:43,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:43,700 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:43,700 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,700 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:43,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:43,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:43,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:43,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:43,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:43,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:43,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:43,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:43,701 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:43,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:43,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,702 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,702 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:43,702 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:43,702 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:43,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:43,702 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:56:43,702 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:56:43,702 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:56:43,702 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:43,702 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:43,702 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,702 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:43,702 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:43,702 DEBUG [Thread-1446] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:56:43,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:43,703 DEBUG [Thread-1446] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:56:43,702 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:43,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:43,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:43,803 DEBUG [Thread-1446] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:56:43,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:56:43,804 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:56:43,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:56:43,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:56:43,805 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:56:43,805 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:43,805 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:56:43,805 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:56:43,806 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:43,806 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:43,806 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,806 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:43,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:43,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:43,807 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,807 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:56:43,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:43,807 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:43,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:56:43,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:43,807 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:56:43,807 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:56:43,807 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:56:43,807 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:56:43,807 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:43,808 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:56:43,808 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:56:43,808 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:56:43,808 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:43,808 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. started... 2023-03-21 11:56:43,808 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. as already flushing 2023-03-21 11:56:43,808 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:43,808 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:56:43,808 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:56:43,808 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,808 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:56:43,808 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:56:43,809 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:43,809 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,809 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:43,809 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:56:43,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:43,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:43,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:43,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:43,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:43,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:43,810 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:56:43,810 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:56:43,810 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@350c52a9[Count = 0] remaining members to acquire global barrier 2023-03-21 11:56:43,810 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:43,811 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:43,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:43,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:43,811 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:56:43,811 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:56:43,811 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:56:43,811 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,811 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:56:43,811 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:56:43,812 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,812 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:43,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:43,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:43,812 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:56:43,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:43,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:43,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:43,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:43,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:43,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:56:43,814 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:56:43,814 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:56:43,814 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:56:43,814 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:56:43,814 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:56:43,814 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:56:43,815 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:43,815 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:43,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:43,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:43,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:43,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:43,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:43,815 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:43,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:43,816 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:43,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:43,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:43,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:43,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:43,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:43,817 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:43,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:43,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,818 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,818 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:43,818 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:43,818 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:43,818 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,818 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:43,818 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:56:43,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:43,818 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:43,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:43,818 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:56:43,818 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:56:43,818 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:43,819 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:43,819 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:43,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:43,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:43,819 DEBUG [Thread-1446] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:56:43,819 DEBUG [Thread-1446] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:56:43,823 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:43,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41826 deadline: 1679399863822, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,823 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:43,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41768 deadline: 1679399863822, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,823 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:43,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41780 deadline: 1679399863822, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,825 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:43,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41744 deadline: 1679399863823, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,825 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:43,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41730 deadline: 1679399863823, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,919 DEBUG [Thread-1446] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:56:43,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:56:43,920 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:56:43,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:56:43,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:56:43,921 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:56:43,921 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:43,921 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:56:43,921 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:56:43,922 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:43,922 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:43,922 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,922 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:43,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:43,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:43,923 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,923 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:56:43,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:43,923 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:43,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:56:43,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:43,923 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:56:43,923 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:56:43,924 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:56:43,924 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:43,924 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:56:43,924 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:56:43,924 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:56:43,924 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:43,924 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. started... 2023-03-21 11:56:43,924 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. as already flushing 2023-03-21 11:56:43,924 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:43,924 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:56:43,924 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:56:43,924 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,924 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:56:43,924 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:56:43,925 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,925 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:43,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:43,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:43,925 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:43,925 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:56:43,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:43,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:43,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:43,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:43,926 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:56:43,926 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d51a477[Count = 0] remaining members to acquire global barrier 2023-03-21 11:56:43,926 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:56:43,926 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:43,927 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:43,927 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:43,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:43,927 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:56:43,927 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:56:43,927 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,927 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:56:43,927 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:56:43,928 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,928 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:56:43,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:43,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:43,928 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:43,928 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:56:43,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:43,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:43,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:43,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:43,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:43,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:56:43,931 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:56:43,931 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:56:43,931 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:56:43,931 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:56:43,931 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:56:43,931 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:56:43,933 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:43,933 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:43,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:43,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:43,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:43,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:43,933 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:43,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:43,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:43,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:43,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:43,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:43,935 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:43,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:43,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:43,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,936 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:43,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:43,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,937 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,937 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:43,937 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:43,937 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:43,937 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:43,937 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:43,937 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:56:43,937 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:56:43,937 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:43,937 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:43,937 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:43,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:43,938 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:43,938 DEBUG [Thread-1446] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:56:43,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:43,938 DEBUG [Thread-1446] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:56:43,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:43,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:44,024 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=48 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/C/152b222df2da49f0a6d33b99bbb9b965 2023-03-21 11:56:44,028 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/A/a89eced45b54415a8c79345f21b0521f as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/a89eced45b54415a8c79345f21b0521f 2023-03-21 11:56:44,032 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/a89eced45b54415a8c79345f21b0521f, entries=200, sequenceid=48, filesize=13.9 K 2023-03-21 11:56:44,033 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/B/68df7fa6a7ce4bb48904c30c851a31c1 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/68df7fa6a7ce4bb48904c30c851a31c1 2023-03-21 11:56:44,038 DEBUG [Thread-1446] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:56:44,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:56:44,040 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:56:44,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:56:44,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:56:44,041 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:56:44,041 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:44,041 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:56:44,041 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:56:44,041 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:44,041 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:44,042 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:44,042 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:44,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:44,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:44,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:44,042 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:44,042 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:56:44,042 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:44,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:56:44,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:44,043 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:56:44,043 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:56:44,043 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:56:44,043 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:56:44,043 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:44,043 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:56:44,043 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:56:44,043 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:56:44,044 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/68df7fa6a7ce4bb48904c30c851a31c1, entries=150, sequenceid=48, filesize=11.6 K 2023-03-21 11:56:44,044 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:44,044 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. started... 2023-03-21 11:56:44,044 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. as already flushing 2023-03-21 11:56:44,044 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:44,044 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:56:44,044 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:56:44,044 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:44,044 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:56:44,046 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:56:44,048 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/C/152b222df2da49f0a6d33b99bbb9b965 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/152b222df2da49f0a6d33b99bbb9b965 2023-03-21 11:56:44,054 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/152b222df2da49f0a6d33b99bbb9b965, entries=150, sequenceid=48, filesize=11.6 K 2023-03-21 11:56:44,054 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=114.05 KB/116790 for 886ee010948b04e81bf5ede3c083ff8b in 858ms, sequenceid=48, compaction requested=true 2023-03-21 11:56:44,055 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 886ee010948b04e81bf5ede3c083ff8b: 2023-03-21 11:56:44,055 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-21 11:56:44,055 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-21 11:56:44,055 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-21 11:56:44,055 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-21 11:56:44,055 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-21 11:56:44,057 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37998 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-21 11:56:44,057 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-21 11:56:44,057 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1912): 886ee010948b04e81bf5ede3c083ff8b/A is initiating minor compaction (all files) 2023-03-21 11:56:44,057 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1912): 886ee010948b04e81bf5ede3c083ff8b/B is initiating minor compaction (all files) 2023-03-21 11:56:44,057 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:44,057 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 886ee010948b04e81bf5ede3c083ff8b/B in TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:44,057 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:44,057 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 886ee010948b04e81bf5ede3c083ff8b/A in TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:44,058 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/56296638322246d2b38dc30711b72646, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/32882af9b9af404ea0e29eba90b740f5, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/68df7fa6a7ce4bb48904c30c851a31c1] into tmpdir=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp, totalSize=34.8 K 2023-03-21 11:56:44,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:44,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:44,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:44,058 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/e563dcb887eb411080fb5ce59692f5ff, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/69ddd8fc4c6c45078ad26db25a236568, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/a89eced45b54415a8c79345f21b0521f] into tmpdir=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp, totalSize=37.1 K 2023-03-21 11:56:44,058 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:44,058 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:56:44,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:44,058 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 56296638322246d2b38dc30711b72646, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1679399802440 2023-03-21 11:56:44,058 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting e563dcb887eb411080fb5ce59692f5ff, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1679399802440 2023-03-21 11:56:44,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:44,058 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 32882af9b9af404ea0e29eba90b740f5, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=35, earliestPutTs=1679399802446 2023-03-21 11:56:44,058 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 69ddd8fc4c6c45078ad26db25a236568, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=35, earliestPutTs=1679399802446 2023-03-21 11:56:44,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:44,059 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 68df7fa6a7ce4bb48904c30c851a31c1, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1679399802584 2023-03-21 11:56:44,059 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting a89eced45b54415a8c79345f21b0521f, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1679399802584 2023-03-21 11:56:44,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:44,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:44,060 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:56:44,060 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d0e2481[Count = 0] remaining members to acquire global barrier 2023-03-21 11:56:44,060 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:56:44,060 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:44,064 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:44,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:44,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:44,064 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:56:44,065 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:56:44,065 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:56:44,065 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:44,065 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:56:44,065 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] throttle.PressureAwareThroughputController(145): 886ee010948b04e81bf5ede3c083ff8b#B#compaction#180 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-21 11:56:44,065 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] throttle.PressureAwareThroughputController(145): 886ee010948b04e81bf5ede3c083ff8b#A#compaction#181 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-21 11:56:44,065 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:56:44,066 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:44,066 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:44,066 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:56:44,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:44,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:44,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:44,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:44,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:44,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:44,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:44,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:44,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:44,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:44,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:56:44,069 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:56:44,069 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:56:44,069 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:56:44,069 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:56:44,069 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:56:44,069 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:56:44,070 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:44,070 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:44,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:44,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:44,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:44,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:44,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:44,070 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:44,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:44,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:44,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:44,071 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:44,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:44,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:44,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:44,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:44,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:44,075 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:44,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:44,077 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:44,078 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:44,078 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:44,078 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:44,078 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:44,078 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:44,078 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:56:44,078 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:56:44,078 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:44,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:44,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:44,078 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:44,078 DEBUG [Thread-1446] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:56:44,078 DEBUG [Thread-1446] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:56:44,078 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:44,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:44,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:44,078 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:56:44,078 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:44,079 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/B/3f862c593eca4303a374927588a54842 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/3f862c593eca4303a374927588a54842 2023-03-21 11:56:44,083 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 886ee010948b04e81bf5ede3c083ff8b/B of 886ee010948b04e81bf5ede3c083ff8b into 3f862c593eca4303a374927588a54842(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-21 11:56:44,084 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 886ee010948b04e81bf5ede3c083ff8b: 2023-03-21 11:56:44,084 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b., storeName=886ee010948b04e81bf5ede3c083ff8b/B, priority=13, startTime=1679399804055; duration=0sec 2023-03-21 11:56:44,084 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-21 11:56:44,084 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-21 11:56:44,084 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-21 11:56:44,085 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1912): 886ee010948b04e81bf5ede3c083ff8b/C is initiating minor compaction (all files) 2023-03-21 11:56:44,085 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 886ee010948b04e81bf5ede3c083ff8b/C in TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:44,085 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/c7b344cea8784e15b7d70f21e807f05a, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/ff9c1d504a974bb3a8aba60a10dbc165, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/152b222df2da49f0a6d33b99bbb9b965] into tmpdir=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp, totalSize=34.8 K 2023-03-21 11:56:44,085 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting c7b344cea8784e15b7d70f21e807f05a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1679399802440 2023-03-21 11:56:44,087 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting ff9c1d504a974bb3a8aba60a10dbc165, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=35, earliestPutTs=1679399802446 2023-03-21 11:56:44,087 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 152b222df2da49f0a6d33b99bbb9b965, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1679399802584 2023-03-21 11:56:44,095 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] throttle.PressureAwareThroughputController(145): 886ee010948b04e81bf5ede3c083ff8b#C#compaction#182 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-21 11:56:44,178 DEBUG [Thread-1446] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:56:44,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:56:44,180 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:56:44,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:56:44,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:56:44,181 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:56:44,181 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:44,181 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:56:44,181 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:56:44,182 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:44,182 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:44,182 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:44,182 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:44,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:44,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:44,183 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:44,183 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:56:44,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:44,183 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:44,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:56:44,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:44,184 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:56:44,184 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:56:44,184 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:56:44,184 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:56:44,184 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:44,184 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:56:44,186 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:56:44,186 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:56:44,186 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:44,186 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. started... 2023-03-21 11:56:44,187 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 886ee010948b04e81bf5ede3c083ff8b 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-21 11:56:44,187 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 886ee010948b04e81bf5ede3c083ff8b, store=A 2023-03-21 11:56:44,187 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:56:44,187 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 886ee010948b04e81bf5ede3c083ff8b, store=B 2023-03-21 11:56:44,187 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:56:44,187 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 886ee010948b04e81bf5ede3c083ff8b, store=C 2023-03-21 11:56:44,187 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:56:44,197 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=69 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/A/3f5ef914f9294db6bdcf5fe357e413d9 2023-03-21 11:56:44,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(9158): Flush requested on 886ee010948b04e81bf5ede3c083ff8b 2023-03-21 11:56:44,326 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. as already flushing 2023-03-21 11:56:44,338 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:44,338 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:44,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41730 deadline: 1679399864332, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:44,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41744 deadline: 1679399864333, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:44,339 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:44,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41826 deadline: 1679399864338, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:44,341 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:44,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41780 deadline: 1679399864339, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:44,342 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:44,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41768 deadline: 1679399864341, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:44,442 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:44,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41730 deadline: 1679399864439, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:44,442 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:44,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41744 deadline: 1679399864439, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:44,442 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:44,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41826 deadline: 1679399864439, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:44,447 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:44,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41768 deadline: 1679399864446, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:44,449 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:44,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41780 deadline: 1679399864447, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:44,482 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/A/66a5c69a5845493ca65368c66fa6dab4 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/66a5c69a5845493ca65368c66fa6dab4 2023-03-21 11:56:44,487 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 886ee010948b04e81bf5ede3c083ff8b/A of 886ee010948b04e81bf5ede3c083ff8b into 66a5c69a5845493ca65368c66fa6dab4(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-21 11:56:44,488 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 886ee010948b04e81bf5ede3c083ff8b: 2023-03-21 11:56:44,488 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b., storeName=886ee010948b04e81bf5ede3c083ff8b/A, priority=13, startTime=1679399804055; duration=0sec 2023-03-21 11:56:44,488 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-21 11:56:44,505 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/C/b2e74ca628444ad3b71837fe5e11bf80 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/b2e74ca628444ad3b71837fe5e11bf80 2023-03-21 11:56:44,514 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 886ee010948b04e81bf5ede3c083ff8b/C of 886ee010948b04e81bf5ede3c083ff8b into b2e74ca628444ad3b71837fe5e11bf80(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-21 11:56:44,514 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 886ee010948b04e81bf5ede3c083ff8b: 2023-03-21 11:56:44,514 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b., storeName=886ee010948b04e81bf5ede3c083ff8b/C, priority=13, startTime=1679399804055; duration=0sec 2023-03-21 11:56:44,514 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-21 11:56:44,610 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=69 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/B/1a406e465c1f45cc92aa7bba0a4a1ca1 2023-03-21 11:56:44,619 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=69 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/C/dc3cd8f6322840d8a4c5637002ae7d47 2023-03-21 11:56:44,623 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/A/3f5ef914f9294db6bdcf5fe357e413d9 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/3f5ef914f9294db6bdcf5fe357e413d9 2023-03-21 11:56:44,626 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/3f5ef914f9294db6bdcf5fe357e413d9, entries=150, sequenceid=69, filesize=11.6 K 2023-03-21 11:56:44,629 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/B/1a406e465c1f45cc92aa7bba0a4a1ca1 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/1a406e465c1f45cc92aa7bba0a4a1ca1 2023-03-21 11:56:44,633 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/1a406e465c1f45cc92aa7bba0a4a1ca1, entries=150, sequenceid=69, filesize=11.6 K 2023-03-21 11:56:44,636 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/C/dc3cd8f6322840d8a4c5637002ae7d47 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/dc3cd8f6322840d8a4c5637002ae7d47 2023-03-21 11:56:44,641 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/dc3cd8f6322840d8a4c5637002ae7d47, entries=150, sequenceid=69, filesize=11.6 K 2023-03-21 11:56:44,642 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 886ee010948b04e81bf5ede3c083ff8b in 456ms, sequenceid=69, compaction requested=false 2023-03-21 11:56:44,642 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 886ee010948b04e81bf5ede3c083ff8b: 2023-03-21 11:56:44,642 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:44,642 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:56:44,642 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:56:44,642 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:44,647 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:56:44,647 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:56:44,651 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:44,651 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:44,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:44,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:44,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:44,652 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:44,652 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:56:44,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:44,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:44,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:44,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:44,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:44,653 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:56:44,653 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5cc36269[Count = 0] remaining members to acquire global barrier 2023-03-21 11:56:44,653 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:56:44,653 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:44,654 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:44,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:44,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:44,654 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:56:44,654 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:56:44,654 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:56:44,654 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:44,654 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:56:44,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(9158): Flush requested on 886ee010948b04e81bf5ede3c083ff8b 2023-03-21 11:56:44,655 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:44,655 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:56:44,655 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:44,655 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:56:44,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:44,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:44,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:44,655 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 886ee010948b04e81bf5ede3c083ff8b 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-21 11:56:44,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:44,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:44,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:44,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:44,657 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 886ee010948b04e81bf5ede3c083ff8b, store=A 2023-03-21 11:56:44,657 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:56:44,657 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 886ee010948b04e81bf5ede3c083ff8b, store=B 2023-03-21 11:56:44,657 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:56:44,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:44,657 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 886ee010948b04e81bf5ede3c083ff8b, store=C 2023-03-21 11:56:44,657 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:56:44,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:44,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:44,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:56:44,659 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:56:44,659 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:56:44,659 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:56:44,659 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:56:44,659 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:56:44,659 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:56:44,666 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:44,666 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:44,666 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:44,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:44,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:44,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:44,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:44,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:44,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:44,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:44,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:44,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:44,668 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:44,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:44,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:44,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:44,669 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:44,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:44,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:44,675 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:44,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:44,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41768 deadline: 1679399864670, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:44,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:44,676 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:44,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41744 deadline: 1679399864671, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:44,676 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:44,676 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:44,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:44,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:44,676 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:44,676 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:44,676 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:44,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41780 deadline: 1679399864671, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:44,677 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:56:44,677 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:56:44,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:44,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:44,676 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:44,677 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:44,677 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:44,677 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:44,677 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:44,677 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:44,677 DEBUG [Thread-1446] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:56:44,677 DEBUG [Thread-1446] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:56:44,678 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:56:44,678 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:44,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41730 deadline: 1679399864675, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:44,680 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:44,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41826 deadline: 1679399864675, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:44,777 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:44,777 DEBUG [Thread-1446] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:56:44,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41768 deadline: 1679399864776, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:44,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:56:44,778 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:44,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41744 deadline: 1679399864777, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:44,779 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:56:44,780 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:44,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41780 deadline: 1679399864777, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:44,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:56:44,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:56:44,780 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:56:44,780 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:44,780 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:56:44,780 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:56:44,781 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:44,781 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:44,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41730 deadline: 1679399864779, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:44,781 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:44,782 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:44,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:44,782 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:44,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:44,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:44,782 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:44,782 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:56:44,782 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:44,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:56:44,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:44,783 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:56:44,785 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:56:44,785 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:56:44,785 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:56:44,785 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:44,785 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:56:44,785 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:56:44,785 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:56:44,785 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:44,785 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. started... 2023-03-21 11:56:44,785 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. as already flushing 2023-03-21 11:56:44,785 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:44,785 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:56:44,786 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:56:44,786 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:44,786 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:44,786 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:56:44,786 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:56:44,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41826 deadline: 1679399864785, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:44,787 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:44,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:44,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:44,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:44,787 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:44,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:44,787 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:44,787 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:56:44,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:44,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:44,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:44,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:44,788 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:56:44,788 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b8dfa42[Count = 0] remaining members to acquire global barrier 2023-03-21 11:56:44,788 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:56:44,788 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:44,789 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:44,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:44,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:44,789 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:56:44,789 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:44,789 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:56:44,789 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:56:44,789 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:56:44,790 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:44,790 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:56:44,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:44,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:44,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:44,790 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:44,790 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:56:44,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:44,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:44,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:44,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:44,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:44,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:44,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:44,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:56:44,791 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:56:44,791 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:56:44,792 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:56:44,792 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:56:44,792 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:56:44,792 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:56:44,792 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:44,792 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:44,792 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:44,792 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:44,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:44,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:44,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:44,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:44,793 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:44,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:44,793 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:44,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:44,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:44,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:44,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:44,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:44,794 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:44,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:44,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:44,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:44,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:44,795 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:44,795 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:44,795 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:44,795 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:44,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:44,795 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:44,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:44,795 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:56:44,795 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:56:44,795 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:56:44,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:44,795 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:44,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:44,796 DEBUG [Thread-1446] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:56:44,796 DEBUG [Thread-1446] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:56:44,796 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:44,796 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:44,796 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:44,796 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:44,896 DEBUG [Thread-1446] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:56:44,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:56:44,897 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:56:44,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:56:44,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:56:44,899 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:56:44,899 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:44,899 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:56:44,899 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:56:44,899 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:44,899 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:44,900 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:44,900 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:44,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:44,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:44,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:44,900 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:44,900 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:44,901 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:56:44,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:56:44,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:44,901 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:56:44,901 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:56:44,901 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:56:44,901 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:56:44,901 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:44,901 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:56:44,902 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:56:44,902 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:56:44,902 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:44,902 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. started... 2023-03-21 11:56:44,902 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. as already flushing 2023-03-21 11:56:44,902 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:44,902 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:56:44,902 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:56:44,902 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:44,902 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:56:44,902 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:56:44,903 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:44,903 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:44,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:44,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:44,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:44,903 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:44,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:44,903 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:56:44,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:44,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:44,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:44,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:44,904 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:56:44,904 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b2f9558[Count = 0] remaining members to acquire global barrier 2023-03-21 11:56:44,904 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:56:44,904 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:44,905 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:44,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:44,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:44,905 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:44,905 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:56:44,905 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:56:44,905 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:56:44,905 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:56:44,906 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:44,906 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:56:44,906 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:44,906 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:56:44,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:44,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:44,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:44,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:44,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:44,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:44,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:44,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:44,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:44,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:44,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:56:44,908 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:56:44,908 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:56:44,908 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:56:44,908 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:56:44,908 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:56:44,908 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:56:44,910 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:44,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:44,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:44,910 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:44,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:44,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:44,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:44,910 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:44,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:44,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:44,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:44,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:44,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:44,911 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:44,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:44,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:44,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:44,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:44,912 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:44,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:44,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:44,913 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:44,913 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:44,913 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:44,913 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:44,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:44,913 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:44,913 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:44,913 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:56:44,913 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:56:44,913 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:56:44,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:44,913 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:44,913 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:44,913 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:44,913 DEBUG [Thread-1446] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:56:44,913 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:44,913 DEBUG [Thread-1446] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:56:44,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:44,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:44,980 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:44,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41768 deadline: 1679399864978, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:44,981 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:44,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41744 deadline: 1679399864979, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:44,981 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:44,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41780 deadline: 1679399864980, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:44,984 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:44,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41730 deadline: 1679399864983, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:44,989 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:44,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41826 deadline: 1679399864988, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:45,014 DEBUG [Thread-1446] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:56:45,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:56:45,015 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:56:45,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:56:45,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:56:45,016 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:56:45,016 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:45,016 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:56:45,016 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:56:45,017 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:45,017 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:45,017 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:45,017 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:45,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:45,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:45,018 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:45,018 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:56:45,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:45,018 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:45,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:56:45,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:45,018 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:56:45,018 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:56:45,018 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:56:45,018 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:56:45,018 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:45,019 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:56:45,019 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:56:45,019 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:56:45,019 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:45,019 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. started... 2023-03-21 11:56:45,019 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. as already flushing 2023-03-21 11:56:45,019 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:45,019 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:56:45,019 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:56:45,019 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:45,019 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:56:45,019 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:56:45,020 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:45,020 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:45,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:45,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:45,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:45,020 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:45,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:45,020 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:56:45,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:45,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:45,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:45,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:45,021 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:56:45,021 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:56:45,021 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@77298890[Count = 0] remaining members to acquire global barrier 2023-03-21 11:56:45,021 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:45,021 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:45,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:45,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:45,022 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:56:45,022 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:45,022 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:56:45,022 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:56:45,022 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:56:45,022 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:56:45,022 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:45,022 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:45,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:45,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:45,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:45,022 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:56:45,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:45,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:45,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:45,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:45,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:45,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:45,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:45,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:56:45,024 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:56:45,024 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:56:45,024 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:56:45,024 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:56:45,024 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:56:45,024 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:56:45,025 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:45,025 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:45,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:45,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:45,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:45,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:45,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:45,025 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:45,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:45,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:45,025 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:45,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:45,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:45,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:45,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:45,026 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:45,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:45,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:45,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:45,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:45,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:45,028 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:45,028 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:45,028 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:45,028 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:45,028 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:56:45,028 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:56:45,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:45,028 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:45,028 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:45,028 DEBUG [Thread-1446] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:56:45,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:45,028 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:45,028 DEBUG [Thread-1446] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:56:45,028 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:56:45,028 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:45,028 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:45,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:45,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:45,028 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:45,103 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=87 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/A/6ec889323a7f4e049f84b9c9e3bca332 2023-03-21 11:56:45,111 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=87 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/B/69d16325d4584f32803a73a8d8966126 2023-03-21 11:56:45,128 DEBUG [Thread-1446] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:56:45,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:56:45,130 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:56:45,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:56:45,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:56:45,131 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:56:45,131 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:45,131 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:56:45,131 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:56:45,131 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:45,131 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:45,132 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:45,132 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:45,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:45,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:45,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:45,133 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:45,133 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:56:45,133 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:45,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:56:45,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:45,133 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:56:45,133 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:56:45,133 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:56:45,133 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:56:45,133 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:45,133 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:56:45,133 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:56:45,133 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:56:45,133 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:45,133 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. started... 2023-03-21 11:56:45,134 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. as already flushing 2023-03-21 11:56:45,134 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:45,134 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:56:45,134 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:56:45,134 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:45,134 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:56:45,134 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:56:45,135 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:45,135 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:45,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:45,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:45,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:45,135 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:45,135 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:56:45,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:45,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:45,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:45,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:45,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:45,136 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:56:45,136 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@303e6c36[Count = 0] remaining members to acquire global barrier 2023-03-21 11:56:45,136 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:56:45,136 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:45,137 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:45,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:45,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:45,137 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:56:45,137 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:56:45,137 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:56:45,137 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:45,137 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:56:45,137 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:45,137 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:56:45,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:45,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:45,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:45,137 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:45,138 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:56:45,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:45,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:45,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:45,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:45,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:45,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:45,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:45,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:56:45,139 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:56:45,139 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:56:45,139 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:56:45,139 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:56:45,139 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:56:45,139 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:56:45,140 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:45,140 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:45,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:45,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:45,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:45,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:45,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:45,140 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:45,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:45,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:45,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:45,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:45,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:45,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:45,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:45,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:45,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:45,141 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:45,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:45,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:45,142 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:45,144 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:45,144 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:45,144 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:45,144 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:45,144 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:45,144 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:56:45,144 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:56:45,144 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:45,144 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:45,144 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:45,144 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:45,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:45,144 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:45,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:45,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:45,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:45,144 DEBUG [Thread-1446] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:56:45,144 DEBUG [Thread-1446] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:56:45,245 DEBUG [Thread-1446] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:56:45,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:56:45,246 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:56:45,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:56:45,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:56:45,247 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:56:45,247 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:45,247 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:56:45,248 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:56:45,248 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:45,248 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:45,248 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:45,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:45,249 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:45,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:45,249 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:45,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:45,249 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:56:45,249 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:45,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:56:45,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:45,249 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:56:45,249 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:56:45,249 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:56:45,249 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:56:45,249 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:45,250 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:56:45,250 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:56:45,250 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:56:45,250 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:45,250 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. started... 2023-03-21 11:56:45,250 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. as already flushing 2023-03-21 11:56:45,250 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:45,250 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:56:45,250 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:56:45,250 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:45,250 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:56:45,250 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:56:45,251 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:45,251 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:45,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:45,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:45,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:45,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:45,251 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:45,251 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:56:45,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:45,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:45,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:45,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:45,252 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:56:45,252 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:56:45,252 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5405df99[Count = 0] remaining members to acquire global barrier 2023-03-21 11:56:45,252 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:45,253 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:45,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:45,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:45,253 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:56:45,253 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:45,253 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:56:45,253 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:56:45,253 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:56:45,253 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:45,253 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:56:45,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:45,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:45,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:45,253 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:45,254 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:56:45,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:45,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:45,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:45,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:45,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:45,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:45,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:45,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:56:45,255 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:56:45,255 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:56:45,255 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:56:45,255 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:56:45,255 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:56:45,255 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:56:45,256 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:45,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:45,256 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:45,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:45,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:45,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:45,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:45,256 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:45,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:45,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:45,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:45,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:45,257 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:45,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:45,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:45,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:45,258 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:45,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:45,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:45,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:45,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:45,260 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:45,260 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:45,260 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:45,260 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:45,260 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:45,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:45,260 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:45,260 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:56:45,260 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:56:45,260 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:45,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:45,260 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:56:45,260 DEBUG [Thread-1446] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:56:45,260 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:45,260 DEBUG [Thread-1446] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:56:45,260 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:45,260 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:45,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:45,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:45,283 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:45,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41768 deadline: 1679399865281, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:45,283 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:45,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41780 deadline: 1679399865282, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:45,283 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:45,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41744 deadline: 1679399865283, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:45,287 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:45,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41730 deadline: 1679399865286, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:45,292 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:45,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41826 deadline: 1679399865291, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:45,360 DEBUG [Thread-1446] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:56:45,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:56:45,362 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:56:45,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:56:45,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:56:45,363 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:56:45,363 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:45,363 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:56:45,363 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:56:45,363 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:45,363 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:45,364 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:45,364 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:45,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:45,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:45,364 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:45,364 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:56:45,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:45,364 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:45,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:56:45,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:45,365 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:56:45,365 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:56:45,365 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:56:45,365 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:56:45,365 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:45,365 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:56:45,365 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:56:45,365 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:56:45,365 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:45,365 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. started... 2023-03-21 11:56:45,365 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. as already flushing 2023-03-21 11:56:45,365 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:45,365 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:56:45,365 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:56:45,365 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:45,365 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:56:45,365 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:56:45,366 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:45,366 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:45,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:45,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:45,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:45,366 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:45,366 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:56:45,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:45,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:45,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:45,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:45,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:45,368 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:56:45,368 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@73270514[Count = 0] remaining members to acquire global barrier 2023-03-21 11:56:45,368 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:56:45,368 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:45,369 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:45,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:45,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:45,369 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:56:45,369 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:56:45,369 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:56:45,369 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:45,369 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:56:45,370 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:45,370 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:56:45,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:45,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:45,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:45,370 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:45,370 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:56:45,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:45,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:45,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:45,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:45,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:45,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:45,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:45,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:56:45,372 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:56:45,372 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:56:45,372 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:56:45,372 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:56:45,372 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:56:45,372 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:56:45,373 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:45,373 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:45,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:45,373 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:45,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:45,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:45,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:45,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:45,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:45,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:45,373 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:45,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:45,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:45,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:45,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:45,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:45,375 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:45,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:45,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:45,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:45,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:45,376 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:45,376 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:45,376 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:45,376 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:45,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:45,376 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:56:45,376 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:45,376 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:56:45,376 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:56:45,376 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:45,376 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:45,376 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:45,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:45,376 DEBUG [Thread-1446] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:56:45,376 DEBUG [Thread-1446] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:56:45,376 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:45,376 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:45,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:45,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:45,442 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-03-21 11:56:45,476 DEBUG [Thread-1446] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:56:45,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:56:45,478 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:56:45,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:56:45,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:56:45,479 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:56:45,479 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:45,479 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:56:45,479 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:56:45,480 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:45,480 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:45,480 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:45,480 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:45,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:45,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:45,481 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:45,481 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:56:45,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:45,481 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:45,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:56:45,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:45,481 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:56:45,481 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:56:45,481 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:56:45,482 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:56:45,482 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:45,482 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:56:45,482 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:56:45,482 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:56:45,482 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:45,482 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. started... 2023-03-21 11:56:45,482 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. as already flushing 2023-03-21 11:56:45,482 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:45,482 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:56:45,482 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:56:45,482 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:45,482 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:56:45,482 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:56:45,483 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:45,483 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:45,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:45,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:45,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:45,484 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:45,484 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:56:45,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:45,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:45,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:45,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:45,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:45,485 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:56:45,485 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@48f17ee8[Count = 0] remaining members to acquire global barrier 2023-03-21 11:56:45,485 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:56:45,485 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:45,486 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:45,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:45,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:45,486 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:56:45,486 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:56:45,486 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:56:45,487 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:45,487 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:56:45,488 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:45,488 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:56:45,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:45,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:45,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:45,488 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:45,488 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:56:45,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:45,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:45,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:45,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:45,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:45,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:45,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:45,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:56:45,490 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:56:45,490 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:56:45,490 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:56:45,490 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:56:45,490 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:56:45,490 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:56:45,491 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:45,491 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:45,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:45,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:45,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:45,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:45,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:45,491 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:45,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:45,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:45,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:45,492 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:45,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:45,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:45,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:45,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:45,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:45,493 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:45,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:45,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:45,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:45,495 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:45,495 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:45,495 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:45,495 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:45,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:45,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:45,495 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:45,495 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:45,495 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:56:45,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:45,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:45,495 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:45,495 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:56:45,495 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:45,495 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:45,495 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:45,495 DEBUG [Thread-1446] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:56:45,495 DEBUG [Thread-1446] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:56:45,521 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=87 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/C/bebb5c9107844c1ab9855109e95ab265 2023-03-21 11:56:45,525 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/A/6ec889323a7f4e049f84b9c9e3bca332 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/6ec889323a7f4e049f84b9c9e3bca332 2023-03-21 11:56:45,528 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/6ec889323a7f4e049f84b9c9e3bca332, entries=150, sequenceid=87, filesize=11.6 K 2023-03-21 11:56:45,529 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/B/69d16325d4584f32803a73a8d8966126 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/69d16325d4584f32803a73a8d8966126 2023-03-21 11:56:45,532 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/69d16325d4584f32803a73a8d8966126, entries=150, sequenceid=87, filesize=11.6 K 2023-03-21 11:56:45,533 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/C/bebb5c9107844c1ab9855109e95ab265 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/bebb5c9107844c1ab9855109e95ab265 2023-03-21 11:56:45,539 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/bebb5c9107844c1ab9855109e95ab265, entries=150, sequenceid=87, filesize=11.6 K 2023-03-21 11:56:45,540 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=93.93 KB/96180 for 886ee010948b04e81bf5ede3c083ff8b in 885ms, sequenceid=87, compaction requested=true 2023-03-21 11:56:45,540 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 886ee010948b04e81bf5ede3c083ff8b: 2023-03-21 11:56:45,540 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-21 11:56:45,540 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-21 11:56:45,540 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-21 11:56:45,540 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-21 11:56:45,540 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-21 11:56:45,541 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35761 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-21 11:56:45,541 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35761 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-21 11:56:45,541 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1912): 886ee010948b04e81bf5ede3c083ff8b/B is initiating minor compaction (all files) 2023-03-21 11:56:45,541 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1912): 886ee010948b04e81bf5ede3c083ff8b/A is initiating minor compaction (all files) 2023-03-21 11:56:45,541 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 886ee010948b04e81bf5ede3c083ff8b/B in TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:45,541 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 886ee010948b04e81bf5ede3c083ff8b/A in TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:45,541 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/3f862c593eca4303a374927588a54842, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/1a406e465c1f45cc92aa7bba0a4a1ca1, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/69d16325d4584f32803a73a8d8966126] into tmpdir=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp, totalSize=34.9 K 2023-03-21 11:56:45,541 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/66a5c69a5845493ca65368c66fa6dab4, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/3f5ef914f9294db6bdcf5fe357e413d9, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/6ec889323a7f4e049f84b9c9e3bca332] into tmpdir=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp, totalSize=34.9 K 2023-03-21 11:56:45,542 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 3f862c593eca4303a374927588a54842, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1679399802584 2023-03-21 11:56:45,542 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 66a5c69a5845493ca65368c66fa6dab4, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1679399802584 2023-03-21 11:56:45,542 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 1a406e465c1f45cc92aa7bba0a4a1ca1, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=69, earliestPutTs=1679399803210 2023-03-21 11:56:45,542 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 3f5ef914f9294db6bdcf5fe357e413d9, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=69, earliestPutTs=1679399803210 2023-03-21 11:56:45,542 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 69d16325d4584f32803a73a8d8966126, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=87, earliestPutTs=1679399804330 2023-03-21 11:56:45,542 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 6ec889323a7f4e049f84b9c9e3bca332, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=87, earliestPutTs=1679399804330 2023-03-21 11:56:45,547 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] throttle.PressureAwareThroughputController(145): 886ee010948b04e81bf5ede3c083ff8b#B#compaction#189 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-21 11:56:45,547 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] throttle.PressureAwareThroughputController(145): 886ee010948b04e81bf5ede3c083ff8b#A#compaction#190 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-21 11:56:45,596 DEBUG [Thread-1446] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:56:45,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:56:45,597 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:56:45,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:56:45,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:56:45,598 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:56:45,598 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:45,598 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:56:45,598 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:56:45,599 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:45,599 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:45,599 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:45,599 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:45,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:45,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:45,600 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:45,600 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:56:45,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:45,600 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:45,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:56:45,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:45,600 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:56:45,600 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:56:45,600 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:56:45,600 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:56:45,600 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:45,600 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:56:45,601 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:56:45,601 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:56:45,601 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:45,601 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. started... 2023-03-21 11:56:45,601 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 886ee010948b04e81bf5ede3c083ff8b 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-03-21 11:56:45,601 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 886ee010948b04e81bf5ede3c083ff8b, store=A 2023-03-21 11:56:45,601 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:56:45,601 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 886ee010948b04e81bf5ede3c083ff8b, store=B 2023-03-21 11:56:45,601 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:56:45,601 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 886ee010948b04e81bf5ede3c083ff8b, store=C 2023-03-21 11:56:45,601 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:56:45,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(9158): Flush requested on 886ee010948b04e81bf5ede3c083ff8b 2023-03-21 11:56:45,785 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. as already flushing 2023-03-21 11:56:45,796 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:45,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41826 deadline: 1679399865793, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:45,797 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:45,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41730 deadline: 1679399865794, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:45,798 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:45,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41744 deadline: 1679399865795, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:45,798 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:45,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41780 deadline: 1679399865795, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:45,798 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:45,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41768 deadline: 1679399865796, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:45,900 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:45,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41730 deadline: 1679399865898, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:45,900 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:45,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41744 deadline: 1679399865898, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:45,900 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:45,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41780 deadline: 1679399865899, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:45,900 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:45,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41768 deadline: 1679399865899, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:45,961 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/A/3f27d1647db74eaeb4ad593602351780 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/3f27d1647db74eaeb4ad593602351780 2023-03-21 11:56:45,964 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/B/408f9df01f7b4a3eb85133064a4f9274 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/408f9df01f7b4a3eb85133064a4f9274 2023-03-21 11:56:45,967 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 886ee010948b04e81bf5ede3c083ff8b/A of 886ee010948b04e81bf5ede3c083ff8b into 3f27d1647db74eaeb4ad593602351780(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-21 11:56:45,967 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 886ee010948b04e81bf5ede3c083ff8b: 2023-03-21 11:56:45,967 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b., storeName=886ee010948b04e81bf5ede3c083ff8b/A, priority=13, startTime=1679399805540; duration=0sec 2023-03-21 11:56:45,967 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-21 11:56:45,967 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-21 11:56:45,970 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35761 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-21 11:56:45,970 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1912): 886ee010948b04e81bf5ede3c083ff8b/C is initiating minor compaction (all files) 2023-03-21 11:56:45,971 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 886ee010948b04e81bf5ede3c083ff8b/C in TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:45,971 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/b2e74ca628444ad3b71837fe5e11bf80, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/dc3cd8f6322840d8a4c5637002ae7d47, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/bebb5c9107844c1ab9855109e95ab265] into tmpdir=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp, totalSize=34.9 K 2023-03-21 11:56:45,971 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting b2e74ca628444ad3b71837fe5e11bf80, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1679399802584 2023-03-21 11:56:45,971 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting dc3cd8f6322840d8a4c5637002ae7d47, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=69, earliestPutTs=1679399803210 2023-03-21 11:56:45,972 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting bebb5c9107844c1ab9855109e95ab265, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=87, earliestPutTs=1679399804330 2023-03-21 11:56:45,973 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 886ee010948b04e81bf5ede3c083ff8b/B of 886ee010948b04e81bf5ede3c083ff8b into 408f9df01f7b4a3eb85133064a4f9274(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-21 11:56:45,973 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 886ee010948b04e81bf5ede3c083ff8b: 2023-03-21 11:56:45,973 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b., storeName=886ee010948b04e81bf5ede3c083ff8b/B, priority=13, startTime=1679399805540; duration=0sec 2023-03-21 11:56:45,973 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-21 11:56:45,985 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] throttle.PressureAwareThroughputController(145): 886ee010948b04e81bf5ede3c083ff8b#C#compaction#192 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-21 11:56:46,018 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=104 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/A/8a76553e26634920bf9f1a3455cd63eb 2023-03-21 11:56:46,039 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/C/2a60838532be43a38197e768a46b94c4 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/2a60838532be43a38197e768a46b94c4 2023-03-21 11:56:46,044 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 886ee010948b04e81bf5ede3c083ff8b/C of 886ee010948b04e81bf5ede3c083ff8b into 2a60838532be43a38197e768a46b94c4(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-21 11:56:46,044 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 886ee010948b04e81bf5ede3c083ff8b: 2023-03-21 11:56:46,044 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b., storeName=886ee010948b04e81bf5ede3c083ff8b/C, priority=13, startTime=1679399805540; duration=0sec 2023-03-21 11:56:46,044 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-21 11:56:46,077 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=104 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/B/495bd7317a7a4ab181d52c9dc37869da 2023-03-21 11:56:46,097 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=104 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/C/fb93bed62d1c42d1bc570f64105a3573 2023-03-21 11:56:46,102 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/A/8a76553e26634920bf9f1a3455cd63eb as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/8a76553e26634920bf9f1a3455cd63eb 2023-03-21 11:56:46,103 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:46,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41744 deadline: 1679399866101, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:46,104 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:46,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41780 deadline: 1679399866102, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:46,104 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:46,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41730 deadline: 1679399866102, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:46,105 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:46,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41768 deadline: 1679399866102, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:46,110 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/8a76553e26634920bf9f1a3455cd63eb, entries=150, sequenceid=104, filesize=11.6 K 2023-03-21 11:56:46,111 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/B/495bd7317a7a4ab181d52c9dc37869da as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/495bd7317a7a4ab181d52c9dc37869da 2023-03-21 11:56:46,118 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/495bd7317a7a4ab181d52c9dc37869da, entries=150, sequenceid=104, filesize=11.6 K 2023-03-21 11:56:46,119 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/C/fb93bed62d1c42d1bc570f64105a3573 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/fb93bed62d1c42d1bc570f64105a3573 2023-03-21 11:56:46,129 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/fb93bed62d1c42d1bc570f64105a3573, entries=150, sequenceid=104, filesize=11.6 K 2023-03-21 11:56:46,129 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 886ee010948b04e81bf5ede3c083ff8b in 528ms, sequenceid=104, compaction requested=false 2023-03-21 11:56:46,130 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 886ee010948b04e81bf5ede3c083ff8b: 2023-03-21 11:56:46,130 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:46,130 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:56:46,130 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:56:46,130 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:46,130 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:56:46,130 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:56:46,131 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:46,131 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:46,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:46,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:46,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:46,131 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:46,131 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:56:46,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:46,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:46,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:46,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:46,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:46,133 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:56:46,133 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b6df41c[Count = 0] remaining members to acquire global barrier 2023-03-21 11:56:46,133 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:56:46,133 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:46,135 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:46,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:46,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:46,135 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:56:46,135 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:56:46,135 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:56:46,135 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:46,135 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:56:46,136 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:56:46,136 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:46,136 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:56:46,136 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:46,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:46,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:46,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:46,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:46,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:46,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:46,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:46,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:46,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:46,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:46,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:56:46,140 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:56:46,140 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:56:46,140 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:56:46,140 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:56:46,140 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:56:46,140 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:56:46,141 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:46,141 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:46,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:46,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:46,141 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:46,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:46,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:46,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:46,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:46,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:46,141 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:46,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:46,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:46,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:46,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:46,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:46,143 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:46,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:46,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:46,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:46,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:46,146 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:46,146 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:46,146 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:46,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:46,146 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:46,146 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:56:46,146 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:56:46,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:46,146 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:46,146 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:46,146 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:46,146 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:46,146 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:46,146 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:46,146 DEBUG [Thread-1446] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:56:46,146 DEBUG [Thread-1446] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:56:46,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:46,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:46,147 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:56:46,246 DEBUG [Thread-1446] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:56:46,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:56:46,248 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:56:46,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:56:46,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:56:46,249 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:56:46,249 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:46,249 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:56:46,249 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:56:46,249 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:46,249 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:46,250 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:46,250 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:46,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:46,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:46,250 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:46,250 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:56:46,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:46,251 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:46,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:56:46,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:46,251 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:56:46,251 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:56:46,251 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:56:46,251 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:56:46,251 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:46,251 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:56:46,251 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:56:46,251 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:56:46,251 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:46,251 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. started... 2023-03-21 11:56:46,252 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 886ee010948b04e81bf5ede3c083ff8b 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-03-21 11:56:46,252 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 886ee010948b04e81bf5ede3c083ff8b, store=A 2023-03-21 11:56:46,252 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:56:46,252 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 886ee010948b04e81bf5ede3c083ff8b, store=B 2023-03-21 11:56:46,252 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:56:46,252 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 886ee010948b04e81bf5ede3c083ff8b, store=C 2023-03-21 11:56:46,252 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:56:46,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(9158): Flush requested on 886ee010948b04e81bf5ede3c083ff8b 2023-03-21 11:56:46,406 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. as already flushing 2023-03-21 11:56:46,418 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:46,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41768 deadline: 1679399866415, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:46,419 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:46,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41744 deadline: 1679399866416, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:46,421 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:46,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41730 deadline: 1679399866418, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:46,421 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:46,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41780 deadline: 1679399866419, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:46,520 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:46,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41768 deadline: 1679399866519, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:46,520 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:46,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41744 deadline: 1679399866520, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:46,524 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:46,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41730 deadline: 1679399866522, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:46,524 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:46,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41780 deadline: 1679399866522, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:46,663 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=122 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/A/f3bcb07e030147e7bdab90dec2de9c99 2023-03-21 11:56:46,723 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:46,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41768 deadline: 1679399866722, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:46,723 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:46,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41744 deadline: 1679399866722, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:46,727 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:46,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41730 deadline: 1679399866725, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:46,727 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:46,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41780 deadline: 1679399866726, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:46,805 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:46,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41826 deadline: 1679399866804, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:47,026 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:47,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41744 deadline: 1679399867025, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:47,026 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:47,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41768 deadline: 1679399867026, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:47,030 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:47,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41730 deadline: 1679399867028, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:47,030 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:47,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41780 deadline: 1679399867029, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:47,076 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=122 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/B/96d379d14ae74735b980252c1771801f 2023-03-21 11:56:47,496 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=122 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/C/1ded539d50604dc4b76062f211fdc006 2023-03-21 11:56:47,499 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/A/f3bcb07e030147e7bdab90dec2de9c99 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/f3bcb07e030147e7bdab90dec2de9c99 2023-03-21 11:56:47,503 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/f3bcb07e030147e7bdab90dec2de9c99, entries=150, sequenceid=122, filesize=11.6 K 2023-03-21 11:56:47,504 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/B/96d379d14ae74735b980252c1771801f as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/96d379d14ae74735b980252c1771801f 2023-03-21 11:56:47,507 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/96d379d14ae74735b980252c1771801f, entries=150, sequenceid=122, filesize=11.6 K 2023-03-21 11:56:47,508 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/C/1ded539d50604dc4b76062f211fdc006 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/1ded539d50604dc4b76062f211fdc006 2023-03-21 11:56:47,521 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/1ded539d50604dc4b76062f211fdc006, entries=150, sequenceid=122, filesize=11.6 K 2023-03-21 11:56:47,522 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 886ee010948b04e81bf5ede3c083ff8b in 1271ms, sequenceid=122, compaction requested=true 2023-03-21 11:56:47,522 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 886ee010948b04e81bf5ede3c083ff8b: 2023-03-21 11:56:47,522 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:47,522 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:56:47,523 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:56:47,523 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:47,526 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:56:47,526 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:56:47,527 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:47,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:47,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:47,527 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:47,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:47,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:47,528 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:47,528 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:56:47,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:47,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:47,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:47,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:47,529 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:56:47,529 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@71ee0bff[Count = 0] remaining members to acquire global barrier 2023-03-21 11:56:47,529 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:56:47,529 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:47,530 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:47,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:47,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:47,530 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:56:47,530 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:56:47,530 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:56:47,530 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:47,530 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:56:47,531 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:47,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(9158): Flush requested on 886ee010948b04e81bf5ede3c083ff8b 2023-03-21 11:56:47,531 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 886ee010948b04e81bf5ede3c083ff8b 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-21 11:56:47,531 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 886ee010948b04e81bf5ede3c083ff8b, store=A 2023-03-21 11:56:47,531 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:56:47,531 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 886ee010948b04e81bf5ede3c083ff8b, store=B 2023-03-21 11:56:47,531 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:56:47,531 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 886ee010948b04e81bf5ede3c083ff8b, store=C 2023-03-21 11:56:47,531 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:56:47,531 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:56:47,531 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:47,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:47,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:47,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:47,531 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:56:47,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:47,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:47,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:47,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:47,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:47,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:47,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:47,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:56:47,535 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:56:47,535 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:56:47,535 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:56:47,535 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:56:47,535 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:56:47,535 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:56:47,549 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:47,549 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:47,549 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:47,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:47,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:47,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:47,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:47,550 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:47,550 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:47,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:47,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:47,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41780 deadline: 1679399867545, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:47,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:47,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:47,551 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:47,551 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:47,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41744 deadline: 1679399867547, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:47,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:47,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:47,551 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:47,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41730 deadline: 1679399867546, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:47,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41768 deadline: 1679399867547, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:47,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:47,552 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:47,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:47,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:47,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:47,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:47,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:47,557 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:47,557 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:47,557 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:47,557 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:47,557 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:56:47,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:47,557 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:47,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:47,558 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:56:47,558 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:47,558 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:56:47,558 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:47,558 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:47,558 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:47,558 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:47,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:47,558 DEBUG [Thread-1446] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:56:47,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:47,558 DEBUG [Thread-1446] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:56:47,652 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:47,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41780 deadline: 1679399867652, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:47,652 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:47,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41744 deadline: 1679399867652, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:47,653 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:47,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41730 deadline: 1679399867652, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:47,655 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:47,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41768 deadline: 1679399867653, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:47,658 DEBUG [Thread-1446] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:56:47,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:56:47,660 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:56:47,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:56:47,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:56:47,661 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:56:47,661 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:47,661 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:56:47,661 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:56:47,661 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:47,661 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:47,662 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:47,662 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:47,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:47,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:47,662 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:47,662 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:56:47,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:47,663 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:47,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:56:47,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:47,663 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:56:47,663 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:56:47,663 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:56:47,663 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:56:47,663 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:47,663 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:56:47,663 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:56:47,663 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:56:47,663 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:47,663 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. started... 2023-03-21 11:56:47,663 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. as already flushing 2023-03-21 11:56:47,664 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:47,664 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:56:47,664 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:56:47,664 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:47,664 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:56:47,664 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:56:47,664 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:47,664 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:47,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:47,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:47,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:47,665 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:47,665 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:56:47,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:47,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:47,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:47,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:47,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:47,666 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:56:47,666 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:56:47,666 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3a2abf73[Count = 0] remaining members to acquire global barrier 2023-03-21 11:56:47,666 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:47,666 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:47,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:47,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:47,666 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:56:47,666 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:56:47,666 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:56:47,666 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:47,666 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:56:47,667 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:47,667 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:56:47,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:47,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:47,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:47,667 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:47,667 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:56:47,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:47,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:47,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:47,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:47,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:47,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:47,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:47,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:56:47,669 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:56:47,669 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:56:47,669 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:56:47,669 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:56:47,669 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:56:47,669 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:56:47,670 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:47,670 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:47,670 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:47,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:47,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:47,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:47,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:47,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:47,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:47,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:47,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:47,670 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:47,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:47,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:47,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:47,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:47,674 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:47,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:47,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:47,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:47,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:47,682 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:47,682 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:47,682 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:47,683 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:47,683 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:47,683 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:56:47,683 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:56:47,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:47,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:47,682 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:47,683 DEBUG [Thread-1446] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:56:47,683 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:56:47,683 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:47,683 DEBUG [Thread-1446] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:56:47,683 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:47,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:47,683 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:47,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:47,683 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:47,783 DEBUG [Thread-1446] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:56:47,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:56:47,785 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:56:47,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:56:47,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:56:47,786 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:56:47,786 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:47,786 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:56:47,786 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:56:47,786 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:47,786 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:47,787 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:47,787 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:47,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:47,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:47,787 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:47,787 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:56:47,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:47,787 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:47,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:56:47,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:47,788 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:56:47,788 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:56:47,788 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:56:47,788 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:56:47,788 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:47,788 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:56:47,788 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:56:47,788 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:56:47,788 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:47,788 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. started... 2023-03-21 11:56:47,788 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. as already flushing 2023-03-21 11:56:47,788 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:47,788 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:56:47,788 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:56:47,788 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:47,789 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:56:47,789 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:56:47,789 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:47,789 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:47,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:47,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:47,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:47,790 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:47,790 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:56:47,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:47,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:47,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:47,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:47,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:47,791 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:56:47,791 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@79ef80e0[Count = 0] remaining members to acquire global barrier 2023-03-21 11:56:47,791 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:56:47,791 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:47,791 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:47,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:47,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:47,792 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:56:47,792 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:56:47,792 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:47,792 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:56:47,792 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:56:47,792 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:47,792 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:56:47,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:47,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:47,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:47,792 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:47,792 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:56:47,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:47,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:47,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:47,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:47,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:47,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:47,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:47,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:56:47,794 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:56:47,794 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:56:47,794 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:56:47,794 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:56:47,794 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:56:47,794 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:56:47,795 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:47,795 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:47,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:47,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:47,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:47,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:47,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:47,795 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:47,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:47,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:47,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:47,796 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:47,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:47,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:47,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:47,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:47,797 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:47,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:47,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:47,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:47,798 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:47,798 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:47,798 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:47,798 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:47,798 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:56:47,798 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:56:47,798 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:47,798 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:47,798 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:47,798 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:47,798 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:47,798 DEBUG [Thread-1446] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:56:47,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:47,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:47,798 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:47,798 DEBUG [Thread-1446] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:56:47,798 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:56:47,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:47,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:47,855 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:47,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41744 deadline: 1679399867853, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:47,856 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:47,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41780 deadline: 1679399867854, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:47,856 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:47,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41730 deadline: 1679399867855, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:47,858 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:47,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41768 deadline: 1679399867855, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:47,899 DEBUG [Thread-1446] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:56:47,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:56:47,901 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:56:47,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:56:47,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:56:47,902 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:56:47,902 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:47,902 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:56:47,902 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:56:47,902 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:47,902 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:47,904 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:47,904 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:47,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:47,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:47,904 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:47,904 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:56:47,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:47,904 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:47,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:56:47,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:47,905 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:56:47,905 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:56:47,905 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:56:47,905 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:56:47,905 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:47,905 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:56:47,905 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:56:47,905 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:56:47,905 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:47,905 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. started... 2023-03-21 11:56:47,905 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. as already flushing 2023-03-21 11:56:47,905 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:47,905 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:56:47,905 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:56:47,905 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:47,905 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:56:47,905 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:56:47,906 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:47,906 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:47,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:47,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:47,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:47,906 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:47,907 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:56:47,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:47,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:47,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:47,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:47,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:47,910 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:56:47,910 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1dc646d5[Count = 0] remaining members to acquire global barrier 2023-03-21 11:56:47,910 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:56:47,910 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:47,914 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:47,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:47,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:47,914 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:56:47,914 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:56:47,914 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:56:47,914 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:47,914 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:56:47,915 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:56:47,915 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:47,915 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:47,915 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:56:47,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:47,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:47,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:47,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:47,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:47,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:47,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:47,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:47,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:47,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:47,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:56:47,916 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:56:47,916 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:56:47,916 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:56:47,916 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:56:47,916 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:56:47,917 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:56:47,917 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:47,917 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:47,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:47,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:47,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:47,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:47,917 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:47,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:47,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:47,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:47,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:47,918 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:47,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:47,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:47,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:47,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:47,919 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:47,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:47,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:47,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:47,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:47,920 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:47,920 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:47,920 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:47,920 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:47,920 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:47,920 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:47,920 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:56:47,920 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:56:47,920 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:56:47,920 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:47,920 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:47,921 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:47,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:47,921 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:47,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:47,921 DEBUG [Thread-1446] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:56:47,921 DEBUG [Thread-1446] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:56:47,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:47,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:47,976 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=141 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/A/cccf067d4c3f46da879c1cd2d5030d9c 2023-03-21 11:56:48,021 DEBUG [Thread-1446] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:56:48,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:56:48,023 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:56:48,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:56:48,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:56:48,024 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:56:48,024 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:48,024 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:56:48,024 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:56:48,024 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:48,024 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:48,025 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,025 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:48,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:48,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:48,025 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,025 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:56:48,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:48,025 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:48,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:56:48,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:48,026 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:56:48,026 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:56:48,026 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:56:48,026 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:56:48,026 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:48,026 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:56:48,026 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:56:48,026 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:56:48,026 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:48,026 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. started... 2023-03-21 11:56:48,026 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. as already flushing 2023-03-21 11:56:48,026 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:48,026 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:56:48,026 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:56:48,026 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,026 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:56:48,026 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:56:48,027 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,027 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:48,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:48,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:48,027 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:48,027 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:56:48,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:48,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:48,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:48,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:48,028 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:56:48,028 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:56:48,028 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1cf60761[Count = 0] remaining members to acquire global barrier 2023-03-21 11:56:48,029 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:48,029 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:48,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:48,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:48,029 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:56:48,029 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:56:48,029 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,029 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:56:48,029 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:56:48,030 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,030 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:56:48,030 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:48,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:48,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:48,030 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:56:48,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:48,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:48,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:48,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:48,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:48,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:56:48,032 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:56:48,032 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:56:48,032 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:56:48,032 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:56:48,032 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:56:48,032 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:56:48,032 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:48,032 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:48,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:48,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:48,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:48,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:48,033 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:48,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:48,033 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:48,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:48,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:48,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:48,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:48,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:48,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:48,034 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:48,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:48,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,036 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:48,036 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:48,036 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:48,036 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:48,036 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:48,036 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:56:48,036 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:56:48,036 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:48,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:48,036 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,036 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:48,036 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:48,036 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:48,036 DEBUG [Thread-1446] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:56:48,036 DEBUG [Thread-1446] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:56:48,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:48,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:48,136 DEBUG [Thread-1446] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:56:48,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:56:48,138 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:56:48,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:56:48,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:56:48,139 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:56:48,139 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:48,139 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:56:48,139 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:56:48,140 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:48,140 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:48,140 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:48,141 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:48,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:48,141 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,141 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:56:48,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:48,141 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:48,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:56:48,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:48,141 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:56:48,142 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:56:48,142 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:56:48,142 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:56:48,142 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:48,142 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:56:48,142 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:56:48,142 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:56:48,142 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:48,142 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. started... 2023-03-21 11:56:48,142 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. as already flushing 2023-03-21 11:56:48,142 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:48,142 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:56:48,142 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:56:48,142 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,142 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:56:48,142 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:56:48,143 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,143 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:48,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:48,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:48,143 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:48,143 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:56:48,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:48,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:48,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:48,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:48,144 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:56:48,144 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7348e2ab[Count = 0] remaining members to acquire global barrier 2023-03-21 11:56:48,144 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:56:48,144 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:48,145 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:48,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:48,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:48,145 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,145 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:56:48,145 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:56:48,145 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:56:48,145 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:56:48,146 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:48,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:48,146 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:56:48,146 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:48,146 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:56:48,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:48,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:48,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:48,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:48,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:48,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:56:48,147 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:56:48,147 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:56:48,147 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:56:48,147 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:56:48,147 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:56:48,147 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:56:48,148 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:48,148 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:48,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:48,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:48,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:48,148 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:48,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:48,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:48,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:48,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:48,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:48,149 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:48,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:48,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:48,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:48,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,150 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:48,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:48,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,151 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,151 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:48,151 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:48,151 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:48,151 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:56:48,151 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:56:48,151 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,151 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:48,151 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:48,151 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:48,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:48,151 DEBUG [Thread-1446] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:56:48,151 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:48,151 DEBUG [Thread-1446] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:56:48,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:48,151 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:48,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:48,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:48,159 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:48,159 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:48,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41780 deadline: 1679399868157, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41730 deadline: 1679399868157, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,159 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:48,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41744 deadline: 1679399868157, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,163 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:48,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41768 deadline: 1679399868161, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,251 DEBUG [Thread-1446] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:56:48,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:56:48,253 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:56:48,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:56:48,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:56:48,254 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:56:48,254 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:48,254 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:56:48,254 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:56:48,254 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:48,254 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:48,255 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:48,255 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:48,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:48,255 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,255 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:56:48,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:48,255 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:48,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:56:48,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:48,256 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:56:48,256 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:56:48,256 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:56:48,256 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:56:48,256 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:48,256 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:56:48,256 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:56:48,256 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:56:48,256 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:48,256 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. started... 2023-03-21 11:56:48,256 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. as already flushing 2023-03-21 11:56:48,256 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:48,256 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:56:48,256 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:56:48,256 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,256 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:56:48,256 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:56:48,258 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,258 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:48,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:48,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:48,258 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:48,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:48,258 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:56:48,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:48,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:48,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:48,259 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:56:48,259 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@762594a2[Count = 0] remaining members to acquire global barrier 2023-03-21 11:56:48,259 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:56:48,259 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:48,260 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:48,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:48,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:48,260 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:56:48,260 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:56:48,260 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,260 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:56:48,260 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:56:48,260 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,260 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:56:48,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:48,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:48,260 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:48,261 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:56:48,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:48,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:48,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:48,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:48,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:48,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:56:48,262 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:56:48,262 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:56:48,262 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:56:48,262 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:56:48,262 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:56:48,262 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:56:48,263 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:48,263 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:48,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:48,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:48,263 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:48,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:48,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:48,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:48,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:48,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:48,264 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:48,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:48,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:48,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:48,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:48,264 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:48,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:48,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,266 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,266 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:48,266 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:48,266 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:48,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:48,266 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:48,266 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:56:48,266 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:56:48,266 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:48,266 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,266 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:48,266 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:48,266 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:48,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:48,266 DEBUG [Thread-1446] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:56:48,266 DEBUG [Thread-1446] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:56:48,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:48,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:48,266 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:56:48,366 DEBUG [Thread-1446] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:56:48,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:56:48,368 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:56:48,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:56:48,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:56:48,369 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:56:48,369 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:48,369 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:56:48,369 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:56:48,370 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:48,370 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:48,370 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:48,370 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:48,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:48,370 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,371 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:56:48,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:48,371 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:48,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:56:48,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:48,371 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:56:48,371 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:56:48,371 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:56:48,371 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:56:48,371 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:48,371 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:56:48,371 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:56:48,371 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:56:48,371 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:48,372 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. started... 2023-03-21 11:56:48,372 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. as already flushing 2023-03-21 11:56:48,372 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:48,372 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:56:48,372 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:56:48,372 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,372 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:56:48,372 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:56:48,372 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,373 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:48,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:48,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:48,373 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:48,373 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:56:48,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:48,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:48,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:48,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:48,374 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:56:48,374 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:56:48,374 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@59fa845d[Count = 0] remaining members to acquire global barrier 2023-03-21 11:56:48,374 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:48,375 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:48,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:48,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:48,375 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:56:48,375 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:56:48,375 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:56:48,375 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,375 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:56:48,375 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:56:48,375 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:48,375 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,375 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:56:48,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:48,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:48,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:48,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:48,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:48,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:48,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:48,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:56:48,377 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:56:48,377 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:56:48,377 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:56:48,377 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:56:48,377 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:56:48,377 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:56:48,378 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:48,378 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:48,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:48,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:48,378 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:48,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:48,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:48,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:48,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:48,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:48,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:48,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:48,379 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:48,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:48,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:48,380 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:48,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:48,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,381 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,381 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:48,381 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:48,381 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:48,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:48,381 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:56:48,381 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:56:48,381 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:48,381 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:48,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:48,381 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:56:48,382 DEBUG [Thread-1446] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:56:48,381 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,382 DEBUG [Thread-1446] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:56:48,382 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:48,382 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:48,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:48,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:48,382 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:48,386 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=141 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/B/5fe1ce0ee1794bbfb9b089dcad916b9b 2023-03-21 11:56:48,482 DEBUG [Thread-1446] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:56:48,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:56:48,484 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:56:48,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:56:48,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:56:48,485 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:56:48,485 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:48,485 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:56:48,485 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:56:48,485 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:48,485 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:48,486 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,486 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:48,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:48,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:48,486 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,486 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:56:48,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:48,486 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:48,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:56:48,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:48,486 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:56:48,487 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:56:48,487 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:56:48,487 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:56:48,487 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:48,487 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:56:48,487 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:56:48,487 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:56:48,487 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:48,487 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. started... 2023-03-21 11:56:48,487 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. as already flushing 2023-03-21 11:56:48,487 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:48,487 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:56:48,487 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:56:48,487 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,487 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:56:48,487 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:56:48,488 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:48,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:48,488 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:48,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:48,488 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:48,488 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:56:48,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:48,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:48,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:48,489 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:56:48,489 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@751b4d0c[Count = 0] remaining members to acquire global barrier 2023-03-21 11:56:48,489 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:56:48,489 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:48,490 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:48,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:48,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:48,490 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:56:48,490 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:56:48,490 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:56:48,490 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,490 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:56:48,491 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,491 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:56:48,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:48,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:48,491 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:48,491 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:56:48,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:48,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:48,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:48,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:48,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:48,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:56:48,492 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:56:48,492 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:56:48,492 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:56:48,492 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:56:48,492 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:56:48,492 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:56:48,493 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:48,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:48,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:48,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:48,493 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:48,493 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:48,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:48,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:48,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:48,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:48,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:48,494 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:48,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:48,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:48,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:48,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,494 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:48,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:48,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,496 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:48,496 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:48,496 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:48,496 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:56:48,496 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:56:48,496 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:48,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:48,496 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:48,496 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:48,496 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,496 DEBUG [Thread-1446] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:56:48,496 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:48,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:48,496 DEBUG [Thread-1446] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:56:48,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:48,496 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:48,496 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:48,596 DEBUG [Thread-1446] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:56:48,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:56:48,598 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:56:48,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:56:48,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:56:48,599 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:56:48,599 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:48,599 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:56:48,599 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:56:48,599 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:48,599 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:48,600 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,600 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:48,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:48,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:48,600 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,600 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:56:48,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:48,600 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:48,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:56:48,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:48,601 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:56:48,601 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:56:48,601 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:56:48,601 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:56:48,601 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:48,601 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:56:48,601 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:56:48,601 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:56:48,601 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:48,601 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. started... 2023-03-21 11:56:48,601 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. as already flushing 2023-03-21 11:56:48,601 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:48,602 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:56:48,602 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:56:48,602 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,602 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:56:48,602 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:56:48,602 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:48,602 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:48,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:48,602 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:48,603 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:56:48,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:48,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:48,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:48,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:48,604 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:56:48,604 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@41f2a4be[Count = 0] remaining members to acquire global barrier 2023-03-21 11:56:48,604 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:56:48,604 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:48,604 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:48,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:48,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:48,604 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,604 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:56:48,604 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:56:48,604 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:56:48,604 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:56:48,606 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:56:48,606 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,606 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:48,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:48,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:48,606 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:56:48,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:48,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:48,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:48,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:48,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:48,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:56:48,607 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:56:48,607 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:56:48,607 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:56:48,607 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:56:48,607 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:56:48,607 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:56:48,608 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:48,608 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:48,608 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:48,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:48,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:48,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:48,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:48,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:48,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:48,609 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:48,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:48,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:48,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:48,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:48,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:48,611 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:48,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:48,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,613 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:48,613 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:48,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:48,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:48,613 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:48,613 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:56:48,613 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:56:48,613 DEBUG [Thread-1446] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:56:48,613 DEBUG [Thread-1446] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:56:48,614 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,614 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:48,614 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:48,614 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,614 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:48,614 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:48,614 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:48,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:48,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:48,615 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:56:48,661 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:48,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41730 deadline: 1679399868660, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,664 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:48,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41780 deadline: 1679399868663, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,664 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:48,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41744 deadline: 1679399868663, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,668 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:48,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41768 deadline: 1679399868667, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,714 DEBUG [Thread-1446] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:56:48,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:56:48,717 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:56:48,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:56:48,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:56:48,718 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:56:48,718 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:48,718 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:56:48,718 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:56:48,718 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:48,718 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:48,719 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:48,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:48,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:48,719 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:48,719 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,719 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:56:48,719 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:48,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:56:48,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:48,720 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:56:48,720 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:56:48,720 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:56:48,720 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:56:48,720 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:48,720 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:56:48,720 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:56:48,720 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:56:48,720 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:48,720 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. started... 2023-03-21 11:56:48,720 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. as already flushing 2023-03-21 11:56:48,721 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:48,721 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:56:48,721 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:56:48,721 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,721 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:56:48,721 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:56:48,721 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:48,721 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:48,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:48,722 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:48,722 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:56:48,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:48,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:48,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:48,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:48,723 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:56:48,723 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b28b8a6[Count = 0] remaining members to acquire global barrier 2023-03-21 11:56:48,723 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:56:48,723 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:48,723 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:48,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:48,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:48,724 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:56:48,724 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:56:48,724 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:56:48,724 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,724 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:56:48,724 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:56:48,724 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,724 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:48,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:48,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:48,724 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:56:48,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:48,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:48,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:48,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:48,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:48,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:56:48,726 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:56:48,726 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:56:48,726 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:56:48,726 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:56:48,726 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:56:48,726 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:56:48,727 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:48,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:48,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:48,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:48,727 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:48,727 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:48,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:48,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:48,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:48,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:48,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:48,728 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:48,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:48,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:48,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:48,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,729 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:48,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:48,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,730 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,730 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:48,730 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:48,730 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:48,730 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:48,730 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:48,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:48,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:48,730 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:56:48,730 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:56:48,730 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,730 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:48,731 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:48,731 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:48,731 DEBUG [Thread-1446] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:56:48,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:48,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:48,731 DEBUG [Thread-1446] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:56:48,795 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=141 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/C/a4d6c61fc44b498b9ba4c6870a1fae71 2023-03-21 11:56:48,798 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/A/cccf067d4c3f46da879c1cd2d5030d9c as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/cccf067d4c3f46da879c1cd2d5030d9c 2023-03-21 11:56:48,801 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/cccf067d4c3f46da879c1cd2d5030d9c, entries=150, sequenceid=141, filesize=11.8 K 2023-03-21 11:56:48,802 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/B/5fe1ce0ee1794bbfb9b089dcad916b9b as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/5fe1ce0ee1794bbfb9b089dcad916b9b 2023-03-21 11:56:48,805 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/5fe1ce0ee1794bbfb9b089dcad916b9b, entries=150, sequenceid=141, filesize=11.8 K 2023-03-21 11:56:48,805 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/C/a4d6c61fc44b498b9ba4c6870a1fae71 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/a4d6c61fc44b498b9ba4c6870a1fae71 2023-03-21 11:56:48,808 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/a4d6c61fc44b498b9ba4c6870a1fae71, entries=150, sequenceid=141, filesize=11.8 K 2023-03-21 11:56:48,809 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 886ee010948b04e81bf5ede3c083ff8b in 1278ms, sequenceid=141, compaction requested=true 2023-03-21 11:56:48,809 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 886ee010948b04e81bf5ede3c083ff8b: 2023-03-21 11:56:48,809 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-21 11:56:48,809 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-21 11:56:48,809 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-21 11:56:48,809 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-21 11:56:48,809 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-21 11:56:48,810 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47900 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-21 11:56:48,810 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1912): 886ee010948b04e81bf5ede3c083ff8b/A is initiating minor compaction (all files) 2023-03-21 11:56:48,810 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47900 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-21 11:56:48,811 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 886ee010948b04e81bf5ede3c083ff8b/A in TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:48,811 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1912): 886ee010948b04e81bf5ede3c083ff8b/B is initiating minor compaction (all files) 2023-03-21 11:56:48,811 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 886ee010948b04e81bf5ede3c083ff8b/B in TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:48,811 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/3f27d1647db74eaeb4ad593602351780, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/8a76553e26634920bf9f1a3455cd63eb, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/f3bcb07e030147e7bdab90dec2de9c99, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/cccf067d4c3f46da879c1cd2d5030d9c] into tmpdir=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp, totalSize=46.8 K 2023-03-21 11:56:48,811 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/408f9df01f7b4a3eb85133064a4f9274, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/495bd7317a7a4ab181d52c9dc37869da, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/96d379d14ae74735b980252c1771801f, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/5fe1ce0ee1794bbfb9b089dcad916b9b] into tmpdir=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp, totalSize=46.8 K 2023-03-21 11:56:48,811 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 3f27d1647db74eaeb4ad593602351780, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=87, earliestPutTs=1679399804330 2023-03-21 11:56:48,811 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 408f9df01f7b4a3eb85133064a4f9274, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=87, earliestPutTs=1679399804330 2023-03-21 11:56:48,811 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 8a76553e26634920bf9f1a3455cd63eb, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=104, earliestPutTs=1679399804669 2023-03-21 11:56:48,811 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 495bd7317a7a4ab181d52c9dc37869da, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=104, earliestPutTs=1679399804669 2023-03-21 11:56:48,812 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting f3bcb07e030147e7bdab90dec2de9c99, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=122, earliestPutTs=1679399805793 2023-03-21 11:56:48,812 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 96d379d14ae74735b980252c1771801f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=122, earliestPutTs=1679399805793 2023-03-21 11:56:48,812 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting cccf067d4c3f46da879c1cd2d5030d9c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=141, earliestPutTs=1679399806417 2023-03-21 11:56:48,812 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 5fe1ce0ee1794bbfb9b089dcad916b9b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=141, earliestPutTs=1679399806417 2023-03-21 11:56:48,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(9158): Flush requested on 886ee010948b04e81bf5ede3c083ff8b 2023-03-21 11:56:48,813 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 886ee010948b04e81bf5ede3c083ff8b 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-03-21 11:56:48,813 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 886ee010948b04e81bf5ede3c083ff8b, store=A 2023-03-21 11:56:48,813 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:56:48,813 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 886ee010948b04e81bf5ede3c083ff8b, store=B 2023-03-21 11:56:48,813 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:56:48,813 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 886ee010948b04e81bf5ede3c083ff8b, store=C 2023-03-21 11:56:48,813 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:56:48,821 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] throttle.PressureAwareThroughputController(145): 886ee010948b04e81bf5ede3c083ff8b#A#compaction#202 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-21 11:56:48,821 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] throttle.PressureAwareThroughputController(145): 886ee010948b04e81bf5ede3c083ff8b#B#compaction#203 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-21 11:56:48,831 DEBUG [Thread-1446] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:56:48,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:56:48,833 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:56:48,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:56:48,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:56:48,834 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:56:48,834 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:48,834 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:56:48,834 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:56:48,834 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:48,834 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:48,835 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:48,835 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:48,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:48,835 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,835 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:56:48,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:48,835 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:48,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:56:48,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:48,836 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:56:48,836 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:56:48,836 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:56:48,836 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:56:48,836 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:48,836 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:56:48,836 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:56:48,836 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:56:48,836 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:48,836 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. started... 2023-03-21 11:56:48,837 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. as already flushing 2023-03-21 11:56:48,837 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:48,837 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:56:48,837 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:56:48,837 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,837 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:56:48,837 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:56:48,838 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,838 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:48,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:48,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:48,838 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:48,838 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:56:48,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:48,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:48,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:48,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:48,840 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:56:48,840 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@167d1ff2[Count = 0] remaining members to acquire global barrier 2023-03-21 11:56:48,840 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:56:48,840 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:48,840 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:48,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:48,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:48,840 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:56:48,840 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:56:48,841 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:56:48,841 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,841 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:56:48,841 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,841 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:56:48,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:48,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:48,841 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:48,841 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:56:48,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:48,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:48,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:48,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:48,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:48,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:56:48,843 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:56:48,843 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:56:48,843 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:56:48,843 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:56:48,843 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:56:48,843 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:56:48,844 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:48,844 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:48,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:48,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:48,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:48,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:48,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:48,844 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:48,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:48,844 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:48,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:48,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:48,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:48,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:48,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:48,845 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:48,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:48,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,847 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,847 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:48,847 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:48,847 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,847 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:48,847 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:56:48,847 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:56:48,847 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:48,847 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:56:48,847 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:48,847 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:48,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:48,847 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:48,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:48,847 DEBUG [Thread-1446] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:56:48,847 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:48,847 DEBUG [Thread-1446] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:56:48,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:48,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:48,867 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:48,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41826 deadline: 1679399868865, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,947 DEBUG [Thread-1446] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:56:48,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:56:48,949 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:56:48,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:56:48,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:56:48,950 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:56:48,950 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:48,950 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:56:48,950 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:56:48,951 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:48,951 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:48,951 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,951 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:48,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:48,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:48,951 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,951 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:56:48,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:48,952 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:48,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:56:48,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:48,952 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:56:48,952 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:56:48,952 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:56:48,952 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:56:48,952 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:48,953 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:56:48,953 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:56:48,953 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:56:48,953 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:48,953 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. started... 2023-03-21 11:56:48,953 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. as already flushing 2023-03-21 11:56:48,953 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:48,953 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:56:48,953 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:56:48,953 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,953 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:56:48,953 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:56:48,954 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,954 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:48,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:48,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:48,954 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:48,954 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:56:48,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:48,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:48,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:48,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:48,956 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:56:48,956 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:56:48,956 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@39a1431c[Count = 0] remaining members to acquire global barrier 2023-03-21 11:56:48,956 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:48,957 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:48,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:48,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:48,957 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:56:48,957 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:56:48,957 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:56:48,957 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,957 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:56:48,957 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,957 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:56:48,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:48,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:48,957 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:48,958 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:56:48,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:48,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:48,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:48,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:48,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:48,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:56:48,959 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:56:48,959 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:56:48,960 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:56:48,960 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:56:48,960 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:56:48,960 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:56:48,961 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:48,961 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:48,961 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:48,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:48,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:48,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:48,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:48,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:48,961 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:48,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:48,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:48,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:48,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:48,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:48,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:48,962 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:48,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:48,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,964 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,964 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:48,964 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:48,964 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:48,964 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:48,964 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:56:48,964 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:56:48,964 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:48,964 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:48,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:48,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:48,964 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:48,964 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:56:48,964 DEBUG [Thread-1446] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:56:48,964 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:48,964 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:48,964 DEBUG [Thread-1446] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:56:48,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:48,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:48,969 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:48,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41826 deadline: 1679399868968, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:49,065 DEBUG [Thread-1446] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:56:49,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:56:49,066 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:56:49,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:56:49,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:56:49,067 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:56:49,067 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:49,067 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:56:49,067 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:56:49,067 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:49,068 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:49,068 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:49,068 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:49,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:49,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:49,068 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:49,068 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:56:49,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:49,069 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:49,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:56:49,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:49,069 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:56:49,069 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:56:49,069 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:56:49,069 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:56:49,069 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:49,069 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:56:49,070 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:56:49,070 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:56:49,070 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:49,070 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. started... 2023-03-21 11:56:49,070 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. as already flushing 2023-03-21 11:56:49,070 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:49,070 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:56:49,070 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:56:49,070 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:49,070 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:56:49,070 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:56:49,071 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:49,071 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:49,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:49,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:49,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:49,072 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:49,072 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:56:49,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:49,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:49,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:49,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:49,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:49,073 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:56:49,073 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6d22d7c2[Count = 0] remaining members to acquire global barrier 2023-03-21 11:56:49,073 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:56:49,073 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:49,073 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:49,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:49,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:49,073 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:56:49,073 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:56:49,074 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:56:49,074 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:49,074 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:56:49,074 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:49,074 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:56:49,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:49,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:49,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:49,074 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:49,074 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:56:49,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:49,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:49,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:49,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:49,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:49,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:49,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:49,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:56:49,076 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:56:49,076 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:56:49,076 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:56:49,076 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:56:49,076 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:56:49,076 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:56:49,077 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:49,077 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:49,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:49,077 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:49,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:49,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:49,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:49,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:49,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:49,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:49,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:49,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:49,078 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:49,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:49,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:49,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:49,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:49,079 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:49,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:49,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:49,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:49,080 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:49,080 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:49,080 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:49,080 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:49,080 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:56:49,080 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:56:49,080 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:49,080 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:49,080 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:49,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:49,080 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:56:49,080 DEBUG [Thread-1446] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:56:49,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:49,080 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:49,080 DEBUG [Thread-1446] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:56:49,081 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:49,081 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:49,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:49,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:49,173 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:49,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41826 deadline: 1679399869171, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:49,181 DEBUG [Thread-1446] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:56:49,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:56:49,182 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:56:49,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:56:49,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:56:49,183 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:56:49,183 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:49,183 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:56:49,183 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:56:49,184 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:49,184 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:49,184 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:49,184 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:49,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:49,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:49,185 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:49,185 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:56:49,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:49,191 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:49,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:56:49,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:49,191 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:56:49,191 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:56:49,191 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:56:49,192 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:56:49,192 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:49,192 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:56:49,192 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:56:49,192 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:56:49,192 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:49,192 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. started... 2023-03-21 11:56:49,192 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. as already flushing 2023-03-21 11:56:49,192 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:49,192 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:56:49,192 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:56:49,192 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:49,192 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:56:49,192 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:56:49,197 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:49,197 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:49,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:49,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:49,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:49,198 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:49,198 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:56:49,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:49,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:49,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:49,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:49,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:49,199 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:56:49,200 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3311af48[Count = 0] remaining members to acquire global barrier 2023-03-21 11:56:49,200 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:56:49,200 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:49,200 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:49,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:49,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:49,200 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:56:49,200 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:56:49,200 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:56:49,200 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:49,200 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:56:49,201 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:56:49,201 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:49,201 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:49,201 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:56:49,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:49,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:49,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:49,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:49,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:49,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:49,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:49,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:49,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:49,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:49,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:56:49,203 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:56:49,203 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:56:49,203 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:56:49,203 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:56:49,203 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:56:49,203 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:56:49,204 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:49,204 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:49,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:49,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:49,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:49,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:49,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:49,204 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:49,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:49,204 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:49,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:49,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:49,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:49,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:49,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:49,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:49,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:49,205 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:49,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:49,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:49,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:49,207 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:49,207 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:49,207 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:49,207 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:49,207 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:49,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:49,207 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:56:49,207 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:56:49,207 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:49,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:49,207 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:49,207 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:49,207 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:49,207 DEBUG [Thread-1446] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:56:49,207 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:49,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:49,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:49,207 DEBUG [Thread-1446] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:56:49,228 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=155 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/A/a33871955a464f2ab68758d689706e38 2023-03-21 11:56:49,236 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/B/3402724f53234b02b7a9a0cf51f14bdf as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/3402724f53234b02b7a9a0cf51f14bdf 2023-03-21 11:56:49,241 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=155 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/B/e9a9f4efe6d5480b89a71c01a862c614 2023-03-21 11:56:49,242 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 886ee010948b04e81bf5ede3c083ff8b/B of 886ee010948b04e81bf5ede3c083ff8b into 3402724f53234b02b7a9a0cf51f14bdf(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-21 11:56:49,242 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 886ee010948b04e81bf5ede3c083ff8b: 2023-03-21 11:56:49,242 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b., storeName=886ee010948b04e81bf5ede3c083ff8b/B, priority=12, startTime=1679399808809; duration=0sec 2023-03-21 11:56:49,243 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-21 11:56:49,243 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-21 11:56:49,245 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47900 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-21 11:56:49,245 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1912): 886ee010948b04e81bf5ede3c083ff8b/C is initiating minor compaction (all files) 2023-03-21 11:56:49,245 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 886ee010948b04e81bf5ede3c083ff8b/C in TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:49,245 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/2a60838532be43a38197e768a46b94c4, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/fb93bed62d1c42d1bc570f64105a3573, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/1ded539d50604dc4b76062f211fdc006, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/a4d6c61fc44b498b9ba4c6870a1fae71] into tmpdir=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp, totalSize=46.8 K 2023-03-21 11:56:49,245 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 2a60838532be43a38197e768a46b94c4, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=87, earliestPutTs=1679399804330 2023-03-21 11:56:49,246 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting fb93bed62d1c42d1bc570f64105a3573, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=104, earliestPutTs=1679399804669 2023-03-21 11:56:49,246 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 1ded539d50604dc4b76062f211fdc006, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=122, earliestPutTs=1679399805793 2023-03-21 11:56:49,246 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting a4d6c61fc44b498b9ba4c6870a1fae71, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=141, earliestPutTs=1679399806417 2023-03-21 11:56:49,265 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/A/44860811be204970af9357742aaead6d as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/44860811be204970af9357742aaead6d 2023-03-21 11:56:49,271 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 886ee010948b04e81bf5ede3c083ff8b/A of 886ee010948b04e81bf5ede3c083ff8b into 44860811be204970af9357742aaead6d(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-21 11:56:49,272 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 886ee010948b04e81bf5ede3c083ff8b: 2023-03-21 11:56:49,272 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b., storeName=886ee010948b04e81bf5ede3c083ff8b/A, priority=12, startTime=1679399808809; duration=0sec 2023-03-21 11:56:49,272 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-21 11:56:49,289 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] throttle.PressureAwareThroughputController(145): 886ee010948b04e81bf5ede3c083ff8b#C#compaction#206 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-21 11:56:49,307 DEBUG [Thread-1446] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:56:49,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:56:49,309 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:56:49,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:56:49,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:56:49,322 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:56:49,322 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:49,322 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:56:49,322 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:56:49,322 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:49,323 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:49,323 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:49,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:49,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:49,323 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:49,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:49,324 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:49,324 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:56:49,324 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:49,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:56:49,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:49,324 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:56:49,335 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:56:49,335 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:56:49,335 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:56:49,335 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:49,335 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:56:49,335 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:56:49,335 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:56:49,335 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:49,336 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. started... 2023-03-21 11:56:49,336 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. as already flushing 2023-03-21 11:56:49,336 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:49,336 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:56:49,336 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:56:49,336 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:49,336 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:56:49,336 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:56:49,337 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:49,337 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:49,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:49,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:49,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:49,338 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:49,338 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:56:49,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:49,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:49,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:49,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:49,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:49,339 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:56:49,339 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d6da457[Count = 0] remaining members to acquire global barrier 2023-03-21 11:56:49,339 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:56:49,339 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:49,339 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:49,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:49,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:49,339 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:56:49,339 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:56:49,339 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:49,339 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:56:49,339 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:56:49,340 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:56:49,340 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:49,340 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:49,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:49,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:49,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:49,340 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:56:49,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:49,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:49,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:49,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:49,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:49,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:49,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:49,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:56:49,342 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:56:49,342 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:56:49,342 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:56:49,342 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:56:49,342 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:56:49,342 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:56:49,344 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:49,344 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:49,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:49,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:49,344 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:49,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:49,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:49,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:49,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:49,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:49,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:49,344 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:49,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:49,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:49,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:49,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:49,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:49,345 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:49,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:49,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:49,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:49,347 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:49,347 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:49,347 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:49,347 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:49,347 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:49,347 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:49,347 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:49,347 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:49,347 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:56:49,347 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:49,347 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:56:49,347 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:56:49,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:49,347 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:49,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:49,347 DEBUG [Thread-1446] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:56:49,347 DEBUG [Thread-1446] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:56:49,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:49,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:49,448 DEBUG [Thread-1446] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:56:49,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:56:49,449 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:56:49,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:56:49,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:56:49,450 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:56:49,450 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:49,450 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:56:49,450 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:56:49,450 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:49,451 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:49,451 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:49,451 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:49,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:49,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:49,451 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:49,451 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:56:49,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:49,452 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:49,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:56:49,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:49,452 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:56:49,452 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:56:49,452 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:56:49,452 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:56:49,452 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:49,452 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:56:49,452 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:56:49,452 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:56:49,453 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:49,453 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. started... 2023-03-21 11:56:49,453 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. as already flushing 2023-03-21 11:56:49,453 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:49,453 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:56:49,453 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:56:49,453 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:49,453 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:56:49,453 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:56:49,454 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:49,454 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:49,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:49,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:49,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:49,454 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:49,454 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:56:49,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:49,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:49,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:49,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:49,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:49,455 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:56:49,455 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7e67a745[Count = 0] remaining members to acquire global barrier 2023-03-21 11:56:49,455 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:56:49,455 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:49,455 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:49,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:49,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:49,456 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:49,456 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:56:49,456 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:56:49,456 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:56:49,456 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:56:49,457 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:49,457 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:56:49,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:49,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:49,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:49,457 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:49,457 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:56:49,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:49,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:49,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:49,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:49,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:49,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:49,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:49,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:56:49,459 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:56:49,459 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:56:49,459 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:56:49,459 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:56:49,459 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:56:49,459 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:56:49,460 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:49,460 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:49,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:49,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:49,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:49,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:49,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:49,460 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:49,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:49,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:49,460 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:49,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:49,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:49,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:49,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:49,461 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:49,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:49,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:49,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:49,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:49,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:49,463 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:49,463 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:49,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:49,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:49,463 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:49,463 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:49,463 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:49,463 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:49,463 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:56:49,463 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:56:49,463 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:49,463 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:49,463 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:49,463 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:49,463 DEBUG [Thread-1446] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:56:49,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:49,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:49,463 DEBUG [Thread-1446] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:56:49,463 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:56:49,475 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:49,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41826 deadline: 1679399869474, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:49,564 DEBUG [Thread-1446] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:56:49,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:56:49,565 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:56:49,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:56:49,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:56:49,566 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:56:49,566 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:49,566 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:56:49,566 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:56:49,567 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:49,567 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:49,567 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:49,567 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:49,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:49,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:49,567 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:49,567 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:56:49,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:49,568 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:49,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:56:49,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:49,568 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:56:49,568 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:56:49,568 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:56:49,568 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:56:49,568 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:49,568 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:56:49,569 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:56:49,569 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:56:49,569 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:49,569 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. started... 2023-03-21 11:56:49,569 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. as already flushing 2023-03-21 11:56:49,569 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:49,570 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:56:49,570 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:56:49,570 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:49,570 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:56:49,570 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:56:49,570 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:49,570 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:49,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:49,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:49,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:49,571 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:49,571 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:56:49,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:49,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:49,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:49,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:49,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:49,572 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:56:49,572 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@500f38c[Count = 0] remaining members to acquire global barrier 2023-03-21 11:56:49,572 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:56:49,572 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:49,572 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:49,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:49,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:49,573 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:56:49,573 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:56:49,573 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:49,573 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:56:49,573 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:56:49,573 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:56:49,573 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:49,573 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:49,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:49,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:49,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:49,573 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:56:49,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:49,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:49,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:49,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:49,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:49,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:49,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:49,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:56:49,575 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:56:49,575 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:56:49,575 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:56:49,575 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:56:49,575 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:56:49,575 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:56:49,576 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:49,576 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:49,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:49,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:49,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:49,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:49,576 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:49,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:49,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:49,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:49,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:49,577 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:49,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:49,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:49,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:49,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:49,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:49,578 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:49,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:49,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:49,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:49,580 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:49,580 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:49,580 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:49,580 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:49,580 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:49,580 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:49,580 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:49,580 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:56:49,580 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:56:49,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:49,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:49,580 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:49,580 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:49,580 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:49,580 DEBUG [Thread-1446] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:56:49,580 DEBUG [Thread-1446] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:56:49,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:49,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:49,674 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:49,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41744 deadline: 1679399869672, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:49,674 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:49,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41780 deadline: 1679399869672, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:49,674 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:49,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41730 deadline: 1679399869672, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:49,676 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:49,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41768 deadline: 1679399869676, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:49,680 DEBUG [Thread-1446] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:56:49,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:56:49,682 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:56:49,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:56:49,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:56:49,683 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:56:49,683 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:49,683 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:56:49,683 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:56:49,683 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:49,683 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:49,684 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:49,684 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:49,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:49,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:49,684 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:49,684 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:56:49,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:49,685 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:49,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:56:49,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:49,685 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:56:49,685 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:56:49,685 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:56:49,685 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:49,685 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:56:49,685 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:56:49,685 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:56:49,685 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:49,685 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. started... 2023-03-21 11:56:49,686 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. as already flushing 2023-03-21 11:56:49,686 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:49,686 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:56:49,686 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:56:49,686 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:49,686 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:56:49,686 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:56:49,687 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:49,687 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:49,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:49,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:49,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:49,687 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:49,687 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:56:49,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:49,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:49,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:49,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:49,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:49,688 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:56:49,688 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:56:49,688 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3bd5678b[Count = 0] remaining members to acquire global barrier 2023-03-21 11:56:49,688 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:49,689 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:49,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:49,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:49,689 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:56:49,689 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:56:49,689 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:56:49,689 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:49,689 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:56:49,690 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:49,690 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:56:49,690 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:49,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:49,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:49,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:49,690 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:56:49,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:49,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:49,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:49,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:49,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:49,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:49,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:49,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:56:49,692 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:56:49,692 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:56:49,692 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:56:49,692 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:56:49,692 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:56:49,692 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:56:49,692 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:49,692 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:49,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:49,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:49,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:49,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:49,693 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:49,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:49,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:49,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:49,693 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:49,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:49,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:49,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:49,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:49,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:49,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:49,694 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:49,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:49,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:49,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:49,696 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:49,696 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:49,696 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:49,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:49,696 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:49,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:49,696 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:56:49,696 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:56:49,696 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:49,696 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:49,696 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:49,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:49,696 DEBUG [Thread-1446] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:56:49,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:49,696 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:49,697 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:49,697 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:49,697 DEBUG [Thread-1446] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:56:49,737 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=155 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/C/6e2232a205484fa7a1e4811e81f53be5 2023-03-21 11:56:49,741 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/C/b1aec5bb7dba4bc6a916c08828fc61c1 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/b1aec5bb7dba4bc6a916c08828fc61c1 2023-03-21 11:56:49,741 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/A/a33871955a464f2ab68758d689706e38 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/a33871955a464f2ab68758d689706e38 2023-03-21 11:56:49,744 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/a33871955a464f2ab68758d689706e38, entries=150, sequenceid=155, filesize=11.8 K 2023-03-21 11:56:49,745 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 886ee010948b04e81bf5ede3c083ff8b/C of 886ee010948b04e81bf5ede3c083ff8b into b1aec5bb7dba4bc6a916c08828fc61c1(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-21 11:56:49,745 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/B/e9a9f4efe6d5480b89a71c01a862c614 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/e9a9f4efe6d5480b89a71c01a862c614 2023-03-21 11:56:49,745 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 886ee010948b04e81bf5ede3c083ff8b: 2023-03-21 11:56:49,745 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b., storeName=886ee010948b04e81bf5ede3c083ff8b/C, priority=12, startTime=1679399808809; duration=0sec 2023-03-21 11:56:49,745 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-21 11:56:49,748 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/e9a9f4efe6d5480b89a71c01a862c614, entries=150, sequenceid=155, filesize=11.8 K 2023-03-21 11:56:49,750 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/C/6e2232a205484fa7a1e4811e81f53be5 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/6e2232a205484fa7a1e4811e81f53be5 2023-03-21 11:56:49,754 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/6e2232a205484fa7a1e4811e81f53be5, entries=150, sequenceid=155, filesize=11.8 K 2023-03-21 11:56:49,754 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 886ee010948b04e81bf5ede3c083ff8b in 941ms, sequenceid=155, compaction requested=false 2023-03-21 11:56:49,754 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 886ee010948b04e81bf5ede3c083ff8b: 2023-03-21 11:56:49,797 DEBUG [Thread-1446] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:56:49,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:56:49,800 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:56:49,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:56:49,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:56:49,801 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:56:49,801 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:49,801 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:56:49,801 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:56:49,801 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:49,801 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:49,802 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:49,802 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:49,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:49,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:49,802 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:49,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:49,802 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:56:49,803 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:49,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:56:49,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:49,803 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:56:49,803 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:56:49,803 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:56:49,803 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:56:49,803 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:49,803 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:56:49,803 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:56:49,803 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:56:49,803 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:49,803 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. started... 2023-03-21 11:56:49,804 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 886ee010948b04e81bf5ede3c083ff8b 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-03-21 11:56:49,804 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 886ee010948b04e81bf5ede3c083ff8b, store=A 2023-03-21 11:56:49,804 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:56:49,804 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 886ee010948b04e81bf5ede3c083ff8b, store=B 2023-03-21 11:56:49,804 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:56:49,804 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 886ee010948b04e81bf5ede3c083ff8b, store=C 2023-03-21 11:56:49,804 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:56:49,827 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=176 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/A/684f34424bb245ffbdb0a6106699be42 2023-03-21 11:56:49,849 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=176 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/B/9648bf22de7046fb9f7cadd8ec1af102 2023-03-21 11:56:49,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(9158): Flush requested on 886ee010948b04e81bf5ede3c083ff8b 2023-03-21 11:56:49,979 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. as already flushing 2023-03-21 11:56:50,012 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:50,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41826 deadline: 1679399870010, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:50,115 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:50,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41826 deadline: 1679399870113, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:50,281 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=176 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/C/86e07301b2664f2997d07456b9227f8c 2023-03-21 11:56:50,284 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/A/684f34424bb245ffbdb0a6106699be42 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/684f34424bb245ffbdb0a6106699be42 2023-03-21 11:56:50,287 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/684f34424bb245ffbdb0a6106699be42, entries=150, sequenceid=176, filesize=11.8 K 2023-03-21 11:56:50,288 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/B/9648bf22de7046fb9f7cadd8ec1af102 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/9648bf22de7046fb9f7cadd8ec1af102 2023-03-21 11:56:50,291 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/9648bf22de7046fb9f7cadd8ec1af102, entries=150, sequenceid=176, filesize=11.8 K 2023-03-21 11:56:50,292 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/C/86e07301b2664f2997d07456b9227f8c as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/86e07301b2664f2997d07456b9227f8c 2023-03-21 11:56:50,295 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/86e07301b2664f2997d07456b9227f8c, entries=150, sequenceid=176, filesize=11.8 K 2023-03-21 11:56:50,296 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 886ee010948b04e81bf5ede3c083ff8b in 493ms, sequenceid=176, compaction requested=true 2023-03-21 11:56:50,296 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 886ee010948b04e81bf5ede3c083ff8b: 2023-03-21 11:56:50,296 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:50,296 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:56:50,296 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:56:50,296 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:50,297 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:56:50,297 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:56:50,298 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:50,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:50,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:50,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:50,298 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:50,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:50,298 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:50,298 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:56:50,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:50,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:50,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:50,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:50,300 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:56:50,300 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:56:50,300 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b3a33ba[Count = 0] remaining members to acquire global barrier 2023-03-21 11:56:50,300 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:50,300 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:50,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:50,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:50,300 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:56:50,300 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:56:50,301 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:56:50,301 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:50,301 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:56:50,301 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:56:50,301 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:50,301 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:50,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:50,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:50,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:50,301 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:56:50,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:50,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:50,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:50,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:50,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:50,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:50,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:50,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:56:50,303 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:56:50,303 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:56:50,303 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:56:50,303 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:56:50,303 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:56:50,303 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:56:50,304 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:50,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:50,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:50,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:50,304 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:50,304 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:50,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:50,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:50,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:50,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:50,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:50,305 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:50,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:50,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:50,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:50,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:50,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:50,306 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:50,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:50,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:50,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:50,308 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:50,308 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:50,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:50,308 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:50,308 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:50,308 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:50,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:50,308 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:50,308 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:50,308 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:50,308 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:50,308 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:50,308 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:56:50,309 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:56:50,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:50,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:50,309 DEBUG [Thread-1446] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:56:50,309 DEBUG [Thread-1446] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:56:50,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(9158): Flush requested on 886ee010948b04e81bf5ede3c083ff8b 2023-03-21 11:56:50,319 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 886ee010948b04e81bf5ede3c083ff8b 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-03-21 11:56:50,319 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 886ee010948b04e81bf5ede3c083ff8b, store=A 2023-03-21 11:56:50,319 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:56:50,320 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 886ee010948b04e81bf5ede3c083ff8b, store=B 2023-03-21 11:56:50,320 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:56:50,320 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 886ee010948b04e81bf5ede3c083ff8b, store=C 2023-03-21 11:56:50,320 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:56:50,365 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:50,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41826 deadline: 1679399870363, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:50,409 DEBUG [Thread-1446] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:56:50,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:56:50,411 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:56:50,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:56:50,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:56:50,412 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:56:50,412 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:50,412 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:56:50,412 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:56:50,412 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:50,412 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:50,413 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:50,413 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:50,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:50,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:50,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:50,413 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:50,413 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:56:50,413 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:50,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:56:50,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:50,414 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:56:50,414 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:56:50,414 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:56:50,414 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:56:50,414 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:50,414 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:56:50,414 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:56:50,414 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:56:50,414 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:50,414 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. started... 2023-03-21 11:56:50,414 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. as already flushing 2023-03-21 11:56:50,415 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:50,415 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:56:50,415 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:56:50,415 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:50,415 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:56:50,415 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:56:50,415 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:50,415 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:50,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:50,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:50,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:50,416 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:50,416 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:56:50,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:50,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:50,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:50,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:50,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:50,417 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:56:50,417 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5824f217[Count = 0] remaining members to acquire global barrier 2023-03-21 11:56:50,417 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:56:50,417 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:50,418 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:50,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:50,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:50,418 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:56:50,418 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:56:50,418 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:50,418 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:56:50,418 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:56:50,419 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:56:50,419 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:50,419 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:50,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:50,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:50,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:50,419 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:56:50,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:50,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:50,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:50,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:50,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:50,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:50,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:50,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:56:50,421 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:56:50,421 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:56:50,421 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:56:50,421 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:56:50,421 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:56:50,421 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:56:50,422 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:50,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:50,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:50,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:50,422 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:50,422 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:50,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:50,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:50,423 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:50,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:50,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:50,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:50,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:50,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:50,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:50,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:50,424 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:50,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:50,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:50,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:50,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:50,425 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:50,425 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:50,425 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:50,425 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:50,425 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:50,425 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:50,425 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:50,425 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:50,425 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:56:50,425 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:50,425 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:56:50,425 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:50,426 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:56:50,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:50,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:50,426 DEBUG [Thread-1446] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:56:50,426 DEBUG [Thread-1446] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:56:50,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:50,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:50,467 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:50,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41826 deadline: 1679399870466, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:50,526 DEBUG [Thread-1446] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:56:50,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:56:50,528 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:56:50,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:56:50,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:56:50,529 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:56:50,529 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:50,529 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:56:50,529 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:56:50,529 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:50,529 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:50,530 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:50,530 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:50,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:50,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:50,530 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:50,530 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:56:50,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:50,530 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:50,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:56:50,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:50,531 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:56:50,531 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:56:50,531 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:56:50,531 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:56:50,531 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:50,531 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:56:50,531 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:56:50,531 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:56:50,531 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:50,531 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. started... 2023-03-21 11:56:50,531 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. as already flushing 2023-03-21 11:56:50,531 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:50,531 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:56:50,532 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:56:50,532 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:50,532 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:56:50,532 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:56:50,532 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:50,532 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:50,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:50,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:50,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:50,533 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:50,533 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:56:50,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:50,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:50,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:50,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:50,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:50,534 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:56:50,534 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a3ae60c[Count = 0] remaining members to acquire global barrier 2023-03-21 11:56:50,534 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:56:50,534 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:50,536 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:50,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:50,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:50,536 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:56:50,536 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:56:50,536 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:56:50,536 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:50,536 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:56:50,537 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:56:50,537 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:50,537 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:50,537 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:56:50,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:50,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:50,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:50,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:50,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:50,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:50,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:50,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:50,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:50,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:50,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:56:50,539 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:56:50,539 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:56:50,539 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:56:50,539 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:56:50,539 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:56:50,539 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:56:50,540 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:50,540 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:50,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:50,540 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:50,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:50,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:50,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:50,540 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:50,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:50,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:50,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:50,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:50,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:50,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:50,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:50,541 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:50,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:50,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:50,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:50,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:50,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:50,543 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:50,543 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:50,543 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:50,543 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:50,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:50,543 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:50,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:50,543 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:50,543 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:50,543 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:50,543 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:50,543 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:50,543 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:56:50,543 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:56:50,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:50,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:50,544 DEBUG [Thread-1446] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:56:50,544 DEBUG [Thread-1446] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:56:50,544 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:56:50,644 DEBUG [Thread-1446] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:56:50,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:56:50,646 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:56:50,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:56:50,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:56:50,647 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:56:50,647 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:50,647 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:56:50,647 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:56:50,647 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:50,647 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:50,648 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:50,648 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:50,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:50,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:50,648 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:50,648 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:56:50,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:50,649 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:50,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:56:50,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:50,649 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:56:50,649 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:56:50,649 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:56:50,649 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:56:50,649 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:50,649 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:56:50,649 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:56:50,649 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:56:50,649 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:50,649 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. started... 2023-03-21 11:56:50,650 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. as already flushing 2023-03-21 11:56:50,650 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:50,650 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:56:50,650 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:56:50,650 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:50,650 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:56:50,650 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:56:50,651 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:50,651 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:50,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:50,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:50,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:50,652 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:50,652 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:56:50,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:50,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:50,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:50,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:50,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:50,653 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:56:50,653 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a433bff[Count = 0] remaining members to acquire global barrier 2023-03-21 11:56:50,653 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:56:50,653 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:50,653 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:50,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:50,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:50,653 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:56:50,653 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:56:50,653 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:56:50,654 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:50,654 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:56:50,654 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:50,654 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:56:50,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:50,654 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:50,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:50,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:50,654 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:56:50,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:50,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:50,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:50,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:50,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:50,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:50,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:50,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:56:50,656 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:56:50,656 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:56:50,656 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:56:50,656 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:56:50,656 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:56:50,656 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:56:50,657 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:50,657 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:50,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:50,657 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:50,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:50,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:50,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:50,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:50,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:50,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:50,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:50,658 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:50,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:50,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:50,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:50,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:50,659 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:50,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:50,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:50,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:50,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:50,661 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:50,661 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:50,661 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:50,661 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:50,661 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:50,661 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:56:50,661 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:56:50,661 DEBUG [Thread-1446] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:56:50,661 DEBUG [Thread-1446] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:56:50,661 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:50,662 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:50,662 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:50,662 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:50,662 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:50,661 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:56:50,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:50,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:50,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:50,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:50,670 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:50,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41826 deadline: 1679399870668, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:50,739 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=191 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/A/229f87cf1f5248bfa2048d5f21ce1592 2023-03-21 11:56:50,761 DEBUG [Thread-1446] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:56:50,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:56:50,763 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:56:50,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:56:50,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:56:50,764 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:56:50,765 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:50,765 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:56:50,765 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:56:50,765 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:50,765 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:50,765 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:50,765 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:50,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:50,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:50,766 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:50,766 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:56:50,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:50,766 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:50,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:56:50,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:50,766 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:56:50,766 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:56:50,766 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:56:50,766 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:56:50,767 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:50,767 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:56:50,767 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:56:50,767 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:56:50,767 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:50,767 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. started... 2023-03-21 11:56:50,767 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. as already flushing 2023-03-21 11:56:50,767 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:50,767 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:56:50,767 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:56:50,767 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:50,767 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:56:50,767 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:56:50,768 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:50,768 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:50,768 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:56:50,768 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:50,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:50,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:50,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:50,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:50,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:50,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:50,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:50,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:50,769 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:56:50,769 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ae66ee[Count = 0] remaining members to acquire global barrier 2023-03-21 11:56:50,769 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:56:50,769 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:50,770 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:50,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:50,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:50,770 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:56:50,770 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:56:50,770 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:56:50,770 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:50,770 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:56:50,771 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:56:50,771 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:50,771 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:50,771 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:56:50,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:50,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:50,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:50,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:50,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:50,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:50,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:50,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:50,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:50,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:50,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:56:50,772 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:56:50,772 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:56:50,772 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:56:50,772 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:56:50,772 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:56:50,772 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:56:50,773 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:50,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:50,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:50,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:50,773 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:50,773 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:50,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:50,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:50,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:50,774 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:50,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:50,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:50,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:50,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:50,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:50,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:50,774 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:50,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:50,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:50,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:50,776 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:50,776 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:56:50,776 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:56:50,776 DEBUG [Thread-1446] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:56:50,776 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:56:50,776 DEBUG [Thread-1446] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:56:50,781 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:50,781 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:50,781 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:50,781 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:50,781 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:50,781 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:50,781 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:50,781 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:50,781 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:50,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:50,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:50,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:50,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:50,876 DEBUG [Thread-1446] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:56:50,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:56:50,878 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:56:50,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:56:50,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:56:50,879 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:56:50,879 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:50,879 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:56:50,879 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:56:50,879 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:50,879 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:50,880 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:50,880 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:50,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:50,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:50,880 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:50,880 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:56:50,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:50,880 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:50,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:56:50,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:50,880 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:56:50,880 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:56:50,881 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:56:50,881 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:56:50,881 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:50,881 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:56:50,881 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:56:50,881 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:56:50,881 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:50,881 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. started... 2023-03-21 11:56:50,881 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. as already flushing 2023-03-21 11:56:50,881 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:50,881 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:56:50,881 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:56:50,881 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:50,881 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:56:50,881 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:56:50,882 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:50,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:50,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:50,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:50,882 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:50,882 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:50,882 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:56:50,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:50,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:50,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:50,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:50,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:50,883 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:56:50,883 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7fe588a6[Count = 0] remaining members to acquire global barrier 2023-03-21 11:56:50,883 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:56:50,883 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:50,884 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:50,884 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:50,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:50,884 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:56:50,884 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:56:50,884 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:50,884 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:56:50,884 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:56:50,884 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:56:50,884 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:50,884 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:50,884 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:56:50,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:50,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:50,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:50,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:50,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:50,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:50,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:50,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:50,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:50,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:50,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:56:50,887 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:56:50,887 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:56:50,887 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:56:50,887 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:56:50,887 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:56:50,887 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:56:50,888 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:50,888 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:50,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:50,888 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:50,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:50,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:50,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:50,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:50,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:50,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:50,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:50,888 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:50,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:50,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:50,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:50,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:50,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:50,889 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:50,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:50,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:50,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:50,890 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:50,890 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:50,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:50,890 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:50,891 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:56:50,891 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:56:50,891 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:50,891 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:50,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:50,891 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:50,891 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:50,891 DEBUG [Thread-1446] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:56:50,891 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:50,891 DEBUG [Thread-1446] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:56:50,891 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:50,891 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:50,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:50,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:50,972 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:50,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41826 deadline: 1679399870971, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:50,991 DEBUG [Thread-1446] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:56:50,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:56:50,993 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:56:50,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:56:50,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:56:50,994 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:56:50,994 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:50,994 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:56:50,994 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:56:50,994 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:50,994 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:50,995 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:50,995 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:50,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:50,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:50,995 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:50,995 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:56:50,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:50,995 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:50,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:56:50,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:50,996 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:56:50,996 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:56:50,996 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:56:50,996 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:56:50,996 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:50,996 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:56:50,996 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:56:50,996 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:56:50,996 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:50,996 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. started... 2023-03-21 11:56:50,996 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. as already flushing 2023-03-21 11:56:50,996 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:50,996 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:56:50,996 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:56:50,996 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:50,996 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:56:50,996 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:56:50,997 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:50,997 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:50,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:50,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:50,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:50,997 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:50,997 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:56:50,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:50,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:50,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:50,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:50,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:50,999 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:56:50,999 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@72620bd0[Count = 0] remaining members to acquire global barrier 2023-03-21 11:56:50,999 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:56:50,999 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:51,000 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:51,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:51,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:51,000 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:56:51,000 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:56:51,000 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:56:51,000 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,000 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:56:51,001 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,001 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:56:51,001 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:51,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:51,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:51,001 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:56:51,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:51,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:51,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:51,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:51,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:51,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:56:51,002 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:56:51,002 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:56:51,002 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:56:51,002 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:56:51,002 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:56:51,002 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:56:51,003 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:51,003 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:51,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:51,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:51,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:51,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:51,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:51,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:51,003 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:51,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:51,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:51,004 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:51,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:51,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:51,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:51,005 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:51,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:51,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,006 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:51,006 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:51,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:51,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:51,006 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,006 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:51,006 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:51,006 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:51,006 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:56:51,006 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,006 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:56:51,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:51,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:51,006 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:56:51,006 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:51,007 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:51,007 DEBUG [Thread-1446] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:56:51,007 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:51,007 DEBUG [Thread-1446] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:56:51,107 DEBUG [Thread-1446] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:56:51,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:56:51,109 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:56:51,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:56:51,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:56:51,110 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:56:51,110 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:51,110 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:56:51,110 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:56:51,110 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:51,110 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:51,111 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,111 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:51,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:51,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:51,111 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,111 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:56:51,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:51,111 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:51,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:56:51,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:51,112 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:56:51,112 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:56:51,112 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:56:51,112 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:56:51,112 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:51,112 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:56:51,112 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:56:51,112 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:56:51,112 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:51,112 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. started... 2023-03-21 11:56:51,112 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. as already flushing 2023-03-21 11:56:51,112 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:51,112 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:56:51,112 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:56:51,112 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,112 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:56:51,112 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:56:51,113 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,113 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:51,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:51,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:51,113 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:51,113 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:56:51,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:51,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:51,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:51,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:51,114 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:56:51,114 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:56:51,114 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@212d4aa0[Count = 0] remaining members to acquire global barrier 2023-03-21 11:56:51,114 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:51,115 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:51,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:51,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:51,115 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:56:51,115 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:56:51,115 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:56:51,115 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,115 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:56:51,115 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:56:51,116 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:51,115 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,116 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:56:51,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:51,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:51,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:51,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:51,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:51,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:51,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:51,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:56:51,117 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:56:51,117 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:56:51,117 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:56:51,117 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:56:51,117 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:56:51,117 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:56:51,118 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:51,118 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:51,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:51,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:51,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:51,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:51,118 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:51,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:51,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:51,119 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:51,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:51,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:51,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:51,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:51,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:51,119 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:51,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:51,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,121 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,121 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:51,121 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:51,121 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,121 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:56:51,121 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:56:51,121 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:51,121 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:51,121 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:51,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:51,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:51,121 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:51,121 DEBUG [Thread-1446] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:56:51,121 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:51,121 DEBUG [Thread-1446] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:56:51,121 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:51,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:51,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:51,121 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:56:51,150 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=191 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/B/c5a054eebc3f4c09b4074604facab818 2023-03-21 11:56:51,221 DEBUG [Thread-1446] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:56:51,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:56:51,223 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:56:51,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:56:51,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:56:51,224 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:56:51,224 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:51,224 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:56:51,224 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:56:51,224 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:51,224 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:51,225 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,225 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:51,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:51,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:51,225 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,225 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:56:51,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:51,225 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:51,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:56:51,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:51,225 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:56:51,226 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:56:51,226 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:56:51,226 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:56:51,226 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:51,226 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:56:51,226 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:56:51,226 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:56:51,226 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:51,226 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. started... 2023-03-21 11:56:51,226 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. as already flushing 2023-03-21 11:56:51,226 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:51,226 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:56:51,226 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:56:51,226 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,226 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:56:51,226 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:56:51,227 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:51,227 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:51,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:51,227 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:51,227 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:56:51,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:51,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:51,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:51,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:51,229 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:56:51,229 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:56:51,229 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5247c17d[Count = 0] remaining members to acquire global barrier 2023-03-21 11:56:51,229 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:51,229 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:51,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:51,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:51,229 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:56:51,229 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:56:51,229 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:56:51,229 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,229 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:56:51,230 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,230 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:56:51,230 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:51,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:51,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:51,230 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:56:51,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:51,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:51,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:51,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:51,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:51,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:56:51,232 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:56:51,232 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:56:51,232 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:56:51,232 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:56:51,232 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:56:51,232 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:56:51,233 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:51,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:51,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:51,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:51,233 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,233 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:51,233 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:51,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:51,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:51,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:51,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:51,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:51,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:51,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:51,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:51,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:51,236 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:51,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:51,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,237 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,237 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:51,237 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:51,237 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,237 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:51,237 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:51,237 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:51,237 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:56:51,237 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:56:51,237 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:56:51,237 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:51,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:51,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:51,237 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:51,237 DEBUG [Thread-1446] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:56:51,237 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:51,237 DEBUG [Thread-1446] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:56:51,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:51,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:51,338 DEBUG [Thread-1446] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:56:51,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:56:51,341 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:56:51,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:56:51,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:56:51,342 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:56:51,342 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:51,342 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:56:51,342 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:56:51,342 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:51,342 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:51,343 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:51,343 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:51,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:51,343 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,343 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:56:51,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:51,343 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:51,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:56:51,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:51,343 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:56:51,344 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:56:51,344 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:56:51,344 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:56:51,344 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:51,344 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:56:51,344 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:56:51,344 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:56:51,344 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:51,344 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. started... 2023-03-21 11:56:51,344 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. as already flushing 2023-03-21 11:56:51,344 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:51,344 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:56:51,344 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:56:51,344 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,344 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:56:51,344 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:56:51,345 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:51,345 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:51,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:51,345 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:51,345 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:56:51,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:51,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:51,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:51,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:51,346 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:56:51,346 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b4c32b[Count = 0] remaining members to acquire global barrier 2023-03-21 11:56:51,346 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:56:51,347 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:51,347 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:51,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:51,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:51,347 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:56:51,347 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,347 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:56:51,347 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:56:51,347 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:56:51,348 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:56:51,348 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,348 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:51,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:51,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:51,349 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:56:51,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:51,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:51,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:51,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:51,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:51,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:56:51,350 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:56:51,350 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:56:51,351 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:56:51,351 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:56:51,351 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:56:51,351 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:56:51,351 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:51,351 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:51,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:51,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:51,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:51,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:51,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:51,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:51,352 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:51,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:51,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:51,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:51,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:51,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:51,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,352 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:51,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:51,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:51,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,354 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,355 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,355 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:51,355 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:51,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:51,355 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:51,355 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:51,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:51,355 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:56:51,355 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:56:51,355 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:51,355 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:51,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:51,355 DEBUG [Thread-1446] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:56:51,355 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:51,355 DEBUG [Thread-1446] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:56:51,355 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:51,355 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:51,456 DEBUG [Thread-1446] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:56:51,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:56:51,457 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:56:51,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:56:51,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:56:51,458 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:56:51,458 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:51,458 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:56:51,458 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:56:51,459 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:51,459 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:51,459 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,459 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:51,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:51,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:51,460 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,460 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:56:51,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:51,460 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:51,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:56:51,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:51,460 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:56:51,461 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:56:51,461 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:56:51,461 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:56:51,461 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:51,461 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:56:51,461 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:56:51,461 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:56:51,461 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:51,461 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. started... 2023-03-21 11:56:51,461 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. as already flushing 2023-03-21 11:56:51,461 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:51,461 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:56:51,461 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:56:51,461 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,461 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:56:51,461 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:56:51,462 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,462 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:51,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:51,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:51,462 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:51,462 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:56:51,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:51,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:51,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:51,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:51,463 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:56:51,463 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:56:51,463 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@636a6446[Count = 0] remaining members to acquire global barrier 2023-03-21 11:56:51,463 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:51,464 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:51,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:51,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:51,464 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,464 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:56:51,464 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:56:51,464 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:56:51,464 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:56:51,465 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,465 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:56:51,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:51,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:51,465 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:51,465 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:56:51,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:51,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:51,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:51,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:51,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:51,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:56:51,466 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:56:51,466 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:56:51,466 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:56:51,466 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:56:51,466 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:56:51,466 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:56:51,467 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:51,467 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:51,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:51,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:51,467 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:51,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:51,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:51,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:51,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:51,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:51,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:51,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:51,468 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:51,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:51,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:51,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,468 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:51,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:51,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,470 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,470 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:51,470 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:51,470 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:51,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:51,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:51,470 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:51,470 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:51,470 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,470 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:56:51,470 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:56:51,470 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:51,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:51,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:51,470 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:51,470 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:51,470 DEBUG [Thread-1446] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:56:51,470 DEBUG [Thread-1446] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:56:51,474 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:51,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41826 deadline: 1679399871473, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,569 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=191 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/C/04ba3c1a363349fd9647273de76344c3 2023-03-21 11:56:51,570 DEBUG [Thread-1446] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:56:51,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:56:51,573 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/A/229f87cf1f5248bfa2048d5f21ce1592 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/229f87cf1f5248bfa2048d5f21ce1592 2023-03-21 11:56:51,573 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:56:51,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:56:51,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:56:51,574 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:56:51,574 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:51,574 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:56:51,574 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:56:51,574 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:51,574 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:51,575 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:51,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:51,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:51,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:51,575 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,576 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:51,576 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,576 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:56:51,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:56:51,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:51,576 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:56:51,577 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:56:51,577 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:56:51,577 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:56:51,577 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:51,577 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:56:51,577 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:56:51,577 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:56:51,577 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:51,577 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. started... 2023-03-21 11:56:51,577 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. as already flushing 2023-03-21 11:56:51,577 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/229f87cf1f5248bfa2048d5f21ce1592, entries=150, sequenceid=191, filesize=11.8 K 2023-03-21 11:56:51,577 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:51,577 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:56:51,577 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:56:51,577 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,577 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:56:51,577 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:56:51,578 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/B/c5a054eebc3f4c09b4074604facab818 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/c5a054eebc3f4c09b4074604facab818 2023-03-21 11:56:51,578 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,578 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:51,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:51,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:51,579 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:51,579 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:56:51,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:51,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:51,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:51,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:51,580 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:56:51,580 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@28595ac1[Count = 0] remaining members to acquire global barrier 2023-03-21 11:56:51,580 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:56:51,580 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:51,580 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:51,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:51,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:51,581 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:56:51,581 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:56:51,581 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:56:51,581 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,581 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:56:51,582 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/c5a054eebc3f4c09b4074604facab818, entries=150, sequenceid=191, filesize=11.8 K 2023-03-21 11:56:51,582 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:56:51,582 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,582 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:51,582 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:56:51,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:51,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:51,582 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/C/04ba3c1a363349fd9647273de76344c3 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/04ba3c1a363349fd9647273de76344c3 2023-03-21 11:56:51,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:51,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:51,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:51,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:51,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:51,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:56:51,584 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:56:51,584 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:56:51,584 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:56:51,584 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:56:51,584 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:56:51,584 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:56:51,585 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:51,585 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:51,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:51,585 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:51,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:51,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:51,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:51,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:51,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:51,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:51,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:51,586 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:51,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:51,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:51,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:51,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,587 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:51,587 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/04ba3c1a363349fd9647273de76344c3, entries=150, sequenceid=191, filesize=11.8 K 2023-03-21 11:56:51,588 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 886ee010948b04e81bf5ede3c083ff8b in 1269ms, sequenceid=191, compaction requested=true 2023-03-21 11:56:51,588 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 886ee010948b04e81bf5ede3c083ff8b: 2023-03-21 11:56:51,588 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-21 11:56:51,588 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-21 11:56:51,588 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-21 11:56:51,588 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-21 11:56:51,588 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-21 11:56:51,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:51,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,590 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:51,590 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:51,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:51,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:51,590 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,590 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:51,590 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:51,590 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,590 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:56:51,590 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:56:51,590 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:51,590 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:51,590 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:51,590 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:51,590 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:56:51,590 DEBUG [Thread-1446] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:56:51,590 DEBUG [Thread-1446] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:56:51,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:51,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:51,598 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48486 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-21 11:56:51,599 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1912): 886ee010948b04e81bf5ede3c083ff8b/A is initiating minor compaction (all files) 2023-03-21 11:56:51,599 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 886ee010948b04e81bf5ede3c083ff8b/A in TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:51,599 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/44860811be204970af9357742aaead6d, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/a33871955a464f2ab68758d689706e38, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/684f34424bb245ffbdb0a6106699be42, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/229f87cf1f5248bfa2048d5f21ce1592] into tmpdir=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp, totalSize=47.3 K 2023-03-21 11:56:51,599 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48486 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-21 11:56:51,599 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 44860811be204970af9357742aaead6d, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=141, earliestPutTs=1679399806417 2023-03-21 11:56:51,599 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1912): 886ee010948b04e81bf5ede3c083ff8b/B is initiating minor compaction (all files) 2023-03-21 11:56:51,600 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 886ee010948b04e81bf5ede3c083ff8b/B in TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:51,600 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/3402724f53234b02b7a9a0cf51f14bdf, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/e9a9f4efe6d5480b89a71c01a862c614, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/9648bf22de7046fb9f7cadd8ec1af102, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/c5a054eebc3f4c09b4074604facab818] into tmpdir=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp, totalSize=47.3 K 2023-03-21 11:56:51,600 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting a33871955a464f2ab68758d689706e38, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=155, earliestPutTs=1679399807541 2023-03-21 11:56:51,600 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 3402724f53234b02b7a9a0cf51f14bdf, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=141, earliestPutTs=1679399806417 2023-03-21 11:56:51,600 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 684f34424bb245ffbdb0a6106699be42, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=176, earliestPutTs=1679399808850 2023-03-21 11:56:51,600 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting e9a9f4efe6d5480b89a71c01a862c614, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=155, earliestPutTs=1679399807541 2023-03-21 11:56:51,600 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 229f87cf1f5248bfa2048d5f21ce1592, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=191, earliestPutTs=1679399810003 2023-03-21 11:56:51,601 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 9648bf22de7046fb9f7cadd8ec1af102, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=176, earliestPutTs=1679399808850 2023-03-21 11:56:51,602 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting c5a054eebc3f4c09b4074604facab818, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=191, earliestPutTs=1679399810003 2023-03-21 11:56:51,631 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] throttle.PressureAwareThroughputController(145): 886ee010948b04e81bf5ede3c083ff8b#A#compaction#213 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-21 11:56:51,644 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] throttle.PressureAwareThroughputController(145): 886ee010948b04e81bf5ede3c083ff8b#B#compaction#214 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-21 11:56:51,668 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/A/aec612a2b8c943feb6f8f623b01737d1 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/aec612a2b8c943feb6f8f623b01737d1 2023-03-21 11:56:51,674 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 886ee010948b04e81bf5ede3c083ff8b/A of 886ee010948b04e81bf5ede3c083ff8b into aec612a2b8c943feb6f8f623b01737d1(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-21 11:56:51,674 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 886ee010948b04e81bf5ede3c083ff8b: 2023-03-21 11:56:51,674 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b., storeName=886ee010948b04e81bf5ede3c083ff8b/A, priority=12, startTime=1679399811588; duration=0sec 2023-03-21 11:56:51,675 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-21 11:56:51,675 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-21 11:56:51,676 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48486 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-21 11:56:51,676 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1912): 886ee010948b04e81bf5ede3c083ff8b/C is initiating minor compaction (all files) 2023-03-21 11:56:51,676 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 886ee010948b04e81bf5ede3c083ff8b/C in TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:51,677 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/b1aec5bb7dba4bc6a916c08828fc61c1, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/6e2232a205484fa7a1e4811e81f53be5, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/86e07301b2664f2997d07456b9227f8c, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/04ba3c1a363349fd9647273de76344c3] into tmpdir=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp, totalSize=47.3 K 2023-03-21 11:56:51,677 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting b1aec5bb7dba4bc6a916c08828fc61c1, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=141, earliestPutTs=1679399806417 2023-03-21 11:56:51,677 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 6e2232a205484fa7a1e4811e81f53be5, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=155, earliestPutTs=1679399807541 2023-03-21 11:56:51,678 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 86e07301b2664f2997d07456b9227f8c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=176, earliestPutTs=1679399808850 2023-03-21 11:56:51,678 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 04ba3c1a363349fd9647273de76344c3, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=191, earliestPutTs=1679399810003 2023-03-21 11:56:51,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(9158): Flush requested on 886ee010948b04e81bf5ede3c083ff8b 2023-03-21 11:56:51,688 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 886ee010948b04e81bf5ede3c083ff8b 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-03-21 11:56:51,688 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 886ee010948b04e81bf5ede3c083ff8b, store=A 2023-03-21 11:56:51,688 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:56:51,688 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 886ee010948b04e81bf5ede3c083ff8b, store=B 2023-03-21 11:56:51,689 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:56:51,689 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 886ee010948b04e81bf5ede3c083ff8b, store=C 2023-03-21 11:56:51,689 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:56:51,690 DEBUG [Thread-1446] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:56:51,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:56:51,693 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:56:51,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:56:51,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:56:51,696 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:56:51,696 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:51,696 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:56:51,696 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:56:51,697 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:51,697 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:51,698 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:51,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:51,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:51,698 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:51,699 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,699 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:56:51,699 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:51,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:56:51,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:51,699 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:56:51,699 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:56:51,699 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:56:51,700 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:56:51,700 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:51,700 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:56:51,700 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/B/53d307cbe7f1425d8bc096bf778af22e as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/53d307cbe7f1425d8bc096bf778af22e 2023-03-21 11:56:51,703 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] throttle.PressureAwareThroughputController(145): 886ee010948b04e81bf5ede3c083ff8b#C#compaction#216 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-21 11:56:51,704 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:56:51,704 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:56:51,704 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:51,704 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. started... 2023-03-21 11:56:51,704 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. as already flushing 2023-03-21 11:56:51,704 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:51,704 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:56:51,704 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:56:51,704 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,705 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:56:51,705 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:56:51,706 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:51,706 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:51,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:51,706 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:51,706 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:56:51,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:51,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:51,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:51,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:51,707 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:56:51,707 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@44f93a9e[Count = 0] remaining members to acquire global barrier 2023-03-21 11:56:51,707 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:56:51,707 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:51,708 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:51,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:51,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:51,708 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:56:51,708 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:56:51,708 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,708 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:56:51,708 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:56:51,709 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 886ee010948b04e81bf5ede3c083ff8b/B of 886ee010948b04e81bf5ede3c083ff8b into 53d307cbe7f1425d8bc096bf778af22e(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-21 11:56:51,709 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 886ee010948b04e81bf5ede3c083ff8b: 2023-03-21 11:56:51,709 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b., storeName=886ee010948b04e81bf5ede3c083ff8b/B, priority=12, startTime=1679399811588; duration=0sec 2023-03-21 11:56:51,709 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-21 11:56:51,709 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,709 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:51,709 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:56:51,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41768 deadline: 1679399871704, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,710 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:51,710 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:56:51,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:51,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:51,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:51,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:51,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:51,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:51,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:51,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:56:51,715 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:56:51,715 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:56:51,715 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:56:51,715 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:56:51,715 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:56:51,715 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:56:51,716 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:51,716 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:51,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:51,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:51,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:51,716 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:51,716 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:51,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:51,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41730 deadline: 1679399871710, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:51,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:51,717 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:51,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:51,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41780 deadline: 1679399871710, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:51,717 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:51,717 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:51,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:51,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41744 deadline: 1679399871710, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:51,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:51,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,718 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:51,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:51,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,721 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:51,721 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:51,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:51,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:51,721 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,721 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:56:51,721 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:56:51,721 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:51,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:51,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:51,721 DEBUG [Thread-1446] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:56:51,721 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:51,721 DEBUG [Thread-1446] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:56:51,721 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:51,721 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,721 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:51,721 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:51,721 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:51,812 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:51,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41768 deadline: 1679399871811, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,818 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:51,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41730 deadline: 1679399871817, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,819 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:51,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41780 deadline: 1679399871817, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,819 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:51,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41744 deadline: 1679399871818, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,821 DEBUG [Thread-1446] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:56:51,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:56:51,823 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:56:51,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:56:51,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:56:51,824 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:56:51,824 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:51,824 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:56:51,824 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:56:51,824 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:51,824 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:51,825 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,825 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:51,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:51,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:51,825 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,825 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:56:51,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:51,825 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:51,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:56:51,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:51,826 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:56:51,826 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:56:51,826 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:56:51,826 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:56:51,826 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:51,826 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:56:51,826 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:56:51,826 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:56:51,826 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:51,826 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. started... 2023-03-21 11:56:51,826 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. as already flushing 2023-03-21 11:56:51,826 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:51,826 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:56:51,826 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:56:51,826 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,827 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:56:51,827 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:56:51,827 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,827 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:51,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:51,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:51,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:51,828 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:51,828 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:56:51,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:51,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:51,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:51,829 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:56:51,829 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@637314b1[Count = 0] remaining members to acquire global barrier 2023-03-21 11:56:51,829 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:56:51,829 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:51,829 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:51,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:51,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:51,829 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:56:51,829 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:56:51,829 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,829 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:56:51,829 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:56:51,830 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,830 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:56:51,830 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:51,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:51,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:51,830 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:56:51,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:51,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:51,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:51,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:51,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:51,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:56:51,832 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:56:51,832 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:56:51,832 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:56:51,832 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:56:51,832 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:56:51,832 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:56:51,833 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:51,833 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:51,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:51,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:51,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:51,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:51,833 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:51,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:51,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:51,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:51,834 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:51,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:51,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:51,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:51,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:51,835 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:51,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:51,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,836 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,836 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:51,836 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:51,836 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:56:51,836 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:51,836 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:51,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:51,836 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:56:51,836 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:51,836 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:56:51,836 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:51,836 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:51,836 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:51,836 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:51,836 DEBUG [Thread-1446] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:56:51,837 DEBUG [Thread-1446] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:56:51,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:51,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:51,937 DEBUG [Thread-1446] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:56:51,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:56:51,941 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:56:51,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:56:51,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:56:51,942 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:56:51,942 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:51,942 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:56:51,942 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:56:51,942 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:51,942 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:51,943 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:51,943 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:51,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:51,943 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,943 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:56:51,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:51,943 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:51,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:56:51,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:51,944 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:56:51,944 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:56:51,944 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:56:51,944 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:56:51,944 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:51,944 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:56:51,944 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:56:51,944 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:56:51,944 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:51,944 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. started... 2023-03-21 11:56:51,944 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. as already flushing 2023-03-21 11:56:51,944 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:51,944 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:56:51,944 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:56:51,944 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,944 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:56:51,944 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:56:51,945 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:51,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:51,946 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:51,946 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:51,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:51,946 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:56:51,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:51,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:51,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:51,947 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:56:51,947 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@530b14cd[Count = 0] remaining members to acquire global barrier 2023-03-21 11:56:51,947 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:56:51,947 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:51,947 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:51,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:51,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:51,948 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:56:51,948 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:56:51,948 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:56:51,948 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,948 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:56:51,948 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,948 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:56:51,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:51,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:51,948 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:51,948 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:56:51,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:51,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:51,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:51,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:51,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:51,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:56:51,950 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:56:51,950 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:56:51,950 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:56:51,950 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:56:51,950 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:56:51,950 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:56:51,951 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:51,951 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:51,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:51,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:51,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:51,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:51,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:51,951 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:51,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:51,951 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:51,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:51,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:51,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:51,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:51,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:51,952 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:51,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:51,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,954 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,954 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:51,954 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:51,954 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:51,954 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:56:51,954 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:56:51,954 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:56:51,954 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:51,954 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:51,954 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:51,954 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:51,954 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:51,954 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:51,954 DEBUG [Thread-1446] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:56:51,954 DEBUG [Thread-1446] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:56:51,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:51,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:51,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:51,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:52,015 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:52,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41768 deadline: 1679399872014, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:52,021 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:52,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41730 deadline: 1679399872019, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:52,021 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:52,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41780 deadline: 1679399872019, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:52,021 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:52,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41744 deadline: 1679399872020, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:52,054 DEBUG [Thread-1446] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:56:52,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:56:52,056 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:56:52,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:56:52,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:56:52,057 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:56:52,057 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:52,057 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:56:52,057 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:56:52,058 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:52,058 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:52,058 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:52,058 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:52,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:52,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:52,058 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:52,058 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:56:52,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:52,059 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:52,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:56:52,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:52,059 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:56:52,059 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:56:52,059 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:56:52,059 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:56:52,059 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:52,059 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:56:52,059 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:56:52,059 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:56:52,059 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:52,060 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. started... 2023-03-21 11:56:52,060 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. as already flushing 2023-03-21 11:56:52,060 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:52,060 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:56:52,060 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:56:52,060 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:52,060 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:56:52,060 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:56:52,061 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:52,061 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:52,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:52,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:52,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:52,061 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:52,061 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:56:52,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:52,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:52,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:52,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:52,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:52,062 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:56:52,062 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3166d33f[Count = 0] remaining members to acquire global barrier 2023-03-21 11:56:52,062 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:56:52,062 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:52,062 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:52,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:52,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:52,063 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:56:52,063 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:52,063 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:56:52,063 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:56:52,063 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:56:52,063 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:52,063 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:56:52,063 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:52,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:52,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:52,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:52,063 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:56:52,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:52,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:52,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:52,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:52,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:52,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:52,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:52,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:56:52,065 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:56:52,065 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:56:52,065 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:56:52,065 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:56:52,065 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:56:52,065 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:56:52,066 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:52,066 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:52,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:52,066 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:52,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:52,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:52,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:52,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:52,066 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:52,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:52,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:52,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:52,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:52,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:52,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:52,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:52,067 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:52,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:52,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:52,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:52,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:52,069 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:52,069 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:52,069 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:52,069 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:52,069 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:52,069 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:52,069 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:56:52,069 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:56:52,069 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:52,069 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:52,069 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:56:52,069 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:52,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:52,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:52,069 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:52,069 DEBUG [Thread-1446] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:56:52,069 DEBUG [Thread-1446] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:56:52,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:52,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:52,112 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=210 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/A/cb558ddc0c4c43e7bd5a417970368420 2023-03-21 11:56:52,118 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/C/1838cea787a5435ea08239055ab37b20 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/1838cea787a5435ea08239055ab37b20 2023-03-21 11:56:52,128 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 886ee010948b04e81bf5ede3c083ff8b/C of 886ee010948b04e81bf5ede3c083ff8b into 1838cea787a5435ea08239055ab37b20(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-21 11:56:52,128 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 886ee010948b04e81bf5ede3c083ff8b: 2023-03-21 11:56:52,128 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b., storeName=886ee010948b04e81bf5ede3c083ff8b/C, priority=12, startTime=1679399811588; duration=0sec 2023-03-21 11:56:52,128 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-21 11:56:52,128 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=210 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/B/12fea09f3a5f4291902c1fbc5262beda 2023-03-21 11:56:52,136 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=210 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/C/7ef9fe3da1e64dd3a63be4130e70ef27 2023-03-21 11:56:52,139 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/A/cb558ddc0c4c43e7bd5a417970368420 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/cb558ddc0c4c43e7bd5a417970368420 2023-03-21 11:56:52,142 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/cb558ddc0c4c43e7bd5a417970368420, entries=150, sequenceid=210, filesize=11.8 K 2023-03-21 11:56:52,143 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/B/12fea09f3a5f4291902c1fbc5262beda as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/12fea09f3a5f4291902c1fbc5262beda 2023-03-21 11:56:52,147 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/12fea09f3a5f4291902c1fbc5262beda, entries=150, sequenceid=210, filesize=11.8 K 2023-03-21 11:56:52,148 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/C/7ef9fe3da1e64dd3a63be4130e70ef27 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/7ef9fe3da1e64dd3a63be4130e70ef27 2023-03-21 11:56:52,157 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/7ef9fe3da1e64dd3a63be4130e70ef27, entries=150, sequenceid=210, filesize=11.8 K 2023-03-21 11:56:52,159 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 886ee010948b04e81bf5ede3c083ff8b in 471ms, sequenceid=210, compaction requested=false 2023-03-21 11:56:52,159 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 886ee010948b04e81bf5ede3c083ff8b: 2023-03-21 11:56:52,169 DEBUG [Thread-1446] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:56:52,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:56:52,172 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:56:52,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:56:52,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:56:52,173 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:56:52,173 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:52,174 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:56:52,174 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:56:52,174 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:52,174 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:52,175 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:52,175 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:52,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:52,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:52,175 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:52,175 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:56:52,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:52,176 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:52,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:56:52,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:52,176 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:56:52,178 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:56:52,178 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:56:52,178 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:56:52,178 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:52,178 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:56:52,182 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:56:52,182 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:56:52,182 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:52,182 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. started... 2023-03-21 11:56:52,182 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 886ee010948b04e81bf5ede3c083ff8b 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-03-21 11:56:52,182 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 886ee010948b04e81bf5ede3c083ff8b, store=A 2023-03-21 11:56:52,182 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:56:52,182 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 886ee010948b04e81bf5ede3c083ff8b, store=B 2023-03-21 11:56:52,182 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:56:52,182 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 886ee010948b04e81bf5ede3c083ff8b, store=C 2023-03-21 11:56:52,182 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:56:52,189 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=226 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/A/18b39d2506944108a539b603f6ecdc84 2023-03-21 11:56:52,200 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=226 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/B/5506afddf73447acaa87276aaf0cb65e 2023-03-21 11:56:52,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(9158): Flush requested on 886ee010948b04e81bf5ede3c083ff8b 2023-03-21 11:56:52,318 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. as already flushing 2023-03-21 11:56:52,335 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:52,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41780 deadline: 1679399872332, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:52,335 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:52,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41768 deadline: 1679399872333, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:52,336 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:52,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41730 deadline: 1679399872333, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:52,337 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:52,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41744 deadline: 1679399872334, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:52,436 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:52,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41780 deadline: 1679399872436, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:52,437 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:52,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41768 deadline: 1679399872436, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:52,438 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:52,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41730 deadline: 1679399872437, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:52,439 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:52,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41744 deadline: 1679399872438, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:52,482 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:52,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41826 deadline: 1679399872480, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:52,615 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=226 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/C/2283fe3ccae44867ae8b2519e280987e 2023-03-21 11:56:52,619 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/A/18b39d2506944108a539b603f6ecdc84 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/18b39d2506944108a539b603f6ecdc84 2023-03-21 11:56:52,623 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/18b39d2506944108a539b603f6ecdc84, entries=150, sequenceid=226, filesize=11.8 K 2023-03-21 11:56:52,624 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/B/5506afddf73447acaa87276aaf0cb65e as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/5506afddf73447acaa87276aaf0cb65e 2023-03-21 11:56:52,627 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/5506afddf73447acaa87276aaf0cb65e, entries=150, sequenceid=226, filesize=11.8 K 2023-03-21 11:56:52,627 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/C/2283fe3ccae44867ae8b2519e280987e as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/2283fe3ccae44867ae8b2519e280987e 2023-03-21 11:56:52,630 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/2283fe3ccae44867ae8b2519e280987e, entries=150, sequenceid=226, filesize=11.8 K 2023-03-21 11:56:52,631 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=107.34 KB/109920 for 886ee010948b04e81bf5ede3c083ff8b in 449ms, sequenceid=226, compaction requested=true 2023-03-21 11:56:52,631 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 886ee010948b04e81bf5ede3c083ff8b: 2023-03-21 11:56:52,631 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:52,631 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:56:52,631 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:56:52,631 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:52,631 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:56:52,631 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:56:52,632 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:52,632 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:52,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:52,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:52,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:52,632 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:52,632 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:56:52,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:52,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:52,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:52,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:52,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:52,635 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:56:52,635 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3c463247[Count = 0] remaining members to acquire global barrier 2023-03-21 11:56:52,635 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:56:52,635 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:52,635 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:52,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:52,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:52,635 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:56:52,636 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:56:52,636 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:56:52,636 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:52,636 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:56:52,636 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:56:52,636 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:52,636 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:52,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:52,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:52,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:52,636 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:56:52,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:52,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:52,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:52,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:52,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:52,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:52,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:52,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:56:52,638 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:56:52,638 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:56:52,638 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:56:52,638 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:56:52,638 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:56:52,638 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:56:52,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(9158): Flush requested on 886ee010948b04e81bf5ede3c083ff8b 2023-03-21 11:56:52,639 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 886ee010948b04e81bf5ede3c083ff8b 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-21 11:56:52,639 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:52,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:52,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:52,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:52,639 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:52,640 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:52,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:52,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:52,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:52,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:52,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:52,640 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:52,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:52,640 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 886ee010948b04e81bf5ede3c083ff8b, store=A 2023-03-21 11:56:52,640 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-21 11:56:52,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:52,640 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 886ee010948b04e81bf5ede3c083ff8b, store=B 2023-03-21 11:56:52,641 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-21 11:56:52,641 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 886ee010948b04e81bf5ede3c083ff8b, store=C 2023-03-21 11:56:52,641 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-21 11:56:52,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:52,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:52,642 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:52,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:52,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:52,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:52,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:52,649 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:52,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41730 deadline: 1679399872647, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:52,649 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:52,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41744 deadline: 1679399872648, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:52,649 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:52,649 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:52,649 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:52,649 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:52,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:52,649 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:56:52,650 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:56:52,649 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:52,650 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:52,650 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:52,650 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:52,650 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:52,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:52,649 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:52,649 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:56:52,650 DEBUG [Thread-1446] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:56:52,650 DEBUG [Thread-1446] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:56:52,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:52,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:52,650 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:52,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41780 deadline: 1679399872648, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:52,651 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:52,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41768 deadline: 1679399872649, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:52,750 DEBUG [Thread-1446] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:56:52,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:56:52,752 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:56:52,752 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:52,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41730 deadline: 1679399872750, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:52,752 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:52,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41744 deadline: 1679399872750, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:52,752 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:52,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41780 deadline: 1679399872751, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:52,752 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:52,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41768 deadline: 1679399872751, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:52,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:56:52,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:56:52,753 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:56:52,753 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:52,753 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:56:52,753 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:56:52,753 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:52,753 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:52,754 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:52,754 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:52,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:52,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:52,754 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:52,754 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:56:52,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:52,755 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:52,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:56:52,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:52,755 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:56:52,755 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:56:52,755 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:56:52,755 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:52,755 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:56:52,756 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:56:52,756 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:56:52,756 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:52,756 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. started... 2023-03-21 11:56:52,756 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. as already flushing 2023-03-21 11:56:52,756 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:52,756 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:56:52,756 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:56:52,756 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:52,756 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:56:52,756 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:56:52,757 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:52,757 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:52,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:52,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:52,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:52,757 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:52,757 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:56:52,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:52,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:52,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:52,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:52,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:52,758 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:56:52,758 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@47eb7fe2[Count = 0] remaining members to acquire global barrier 2023-03-21 11:56:52,758 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:56:52,758 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:52,758 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:52,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:52,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:52,758 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:52,759 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:56:52,758 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:56:52,759 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:56:52,759 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:56:52,759 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:52,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:52,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:52,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:52,759 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:56:52,759 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:52,759 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:56:52,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:52,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:52,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:52,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:52,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:52,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:52,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:52,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:56:52,761 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:56:52,761 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:56:52,761 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:56:52,761 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:56:52,761 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:56:52,761 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:56:52,762 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:52,763 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:52,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:52,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:52,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:52,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:52,763 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:52,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:52,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:52,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:52,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:52,763 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:52,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:52,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:52,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:52,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:52,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:52,764 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:52,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:52,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:52,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:52,765 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:52,765 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:52,765 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:52,765 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:52,765 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:52,765 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:52,765 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:56:52,765 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:56:52,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:52,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:52,765 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:52,765 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:52,766 DEBUG [Thread-1446] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:56:52,766 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:52,766 DEBUG [Thread-1446] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:56:52,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:52,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:52,766 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:52,866 DEBUG [Thread-1446] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:56:52,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:56:52,868 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:56:52,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:56:52,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:56:52,869 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:56:52,869 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:52,869 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:56:52,869 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:56:52,869 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:52,869 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:52,870 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:52,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:52,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:52,870 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:52,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:52,874 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:52,874 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:56:52,874 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:52,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:56:52,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:52,874 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:56:52,875 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:56:52,875 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:56:52,875 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:56:52,875 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:52,875 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:56:52,875 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:56:52,875 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:56:52,875 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:52,875 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. started... 2023-03-21 11:56:52,875 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. as already flushing 2023-03-21 11:56:52,875 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:52,875 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:56:52,875 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:56:52,875 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:52,876 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:56:52,876 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:56:52,876 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:52,876 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:52,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:52,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:52,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:52,877 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:52,877 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:56:52,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:52,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:52,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:52,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:52,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:52,878 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:56:52,878 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4ccda6ae[Count = 0] remaining members to acquire global barrier 2023-03-21 11:56:52,878 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:56:52,878 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:52,879 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:52,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:52,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:52,879 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:56:52,879 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:56:52,879 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:56:52,879 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:52,879 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:56:52,882 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:56:52,882 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:52,882 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:52,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:52,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:52,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:52,882 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:56:52,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:52,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:52,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:52,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:52,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:52,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:52,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:52,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:56:52,884 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:56:52,885 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:56:52,885 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:56:52,885 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:56:52,885 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:56:52,885 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:56:52,886 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:52,886 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:52,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:52,886 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:52,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:52,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:52,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:52,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:52,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:52,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:52,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:52,887 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:52,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:52,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:52,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:52,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:52,888 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:52,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:52,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:52,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:52,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:52,890 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:52,890 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:52,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:52,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:52,890 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:52,890 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:52,890 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:52,890 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:52,890 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:52,890 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:52,890 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:52,891 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:52,891 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:56:52,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:52,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:52,891 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:56:52,891 DEBUG [Thread-1446] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:56:52,891 DEBUG [Thread-1446] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:56:52,891 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:56:52,955 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:52,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41730 deadline: 1679399872953, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:52,957 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:52,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41780 deadline: 1679399872953, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:52,957 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:52,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41744 deadline: 1679399872954, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:52,957 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:52,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41768 deadline: 1679399872954, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:52,991 DEBUG [Thread-1446] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:56:52,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:56:52,993 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:56:52,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:56:52,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:56:52,994 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:56:52,994 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:52,994 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:56:52,994 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:56:52,994 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:52,994 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:52,996 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:52,996 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:52,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:52,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:52,996 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:52,996 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:56:52,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:52,996 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:52,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:56:52,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:52,997 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:56:52,997 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:56:52,997 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:56:52,997 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:56:52,997 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:52,997 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:56:52,997 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:56:52,997 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:56:52,997 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:52,997 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. started... 2023-03-21 11:56:52,997 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. as already flushing 2023-03-21 11:56:52,997 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:52,997 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:56:52,997 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:56:52,997 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:52,997 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:56:52,997 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:56:52,998 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:52,998 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:52,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:52,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:52,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:52,998 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:52,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:52,999 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:56:52,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:52,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:52,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:52,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:52,999 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:56:52,999 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b356b23[Count = 0] remaining members to acquire global barrier 2023-03-21 11:56:52,999 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:56:52,999 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:53,000 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:53,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:53,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:53,000 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:56:53,000 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:56:53,000 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:53,000 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:56:53,000 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:56:53,001 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:53,001 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:56:53,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:53,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:53,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:53,001 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:53,001 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:56:53,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:53,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:53,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:53,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:53,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:53,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:53,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:53,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:56:53,002 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:56:53,002 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:56:53,002 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:56:53,002 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:56:53,002 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:56:53,002 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:56:53,003 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:53,003 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:53,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:53,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:53,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:53,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:53,004 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:53,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:53,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:53,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:53,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:53,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:53,004 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:53,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:53,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:53,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:53,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:53,005 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:53,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:53,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:53,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:53,006 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:53,006 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:53,006 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:53,007 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:53,007 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:53,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:53,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:53,007 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:56:53,007 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:56:53,007 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:53,007 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:56:53,007 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:53,007 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:53,007 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:53,007 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:53,007 DEBUG [Thread-1446] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:56:53,007 DEBUG [Thread-1446] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:56:53,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:53,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:53,068 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=247 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/A/a88afc7dc59744b2a8975118df873fed 2023-03-21 11:56:53,107 DEBUG [Thread-1446] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:56:53,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:56:53,109 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:56:53,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:56:53,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:56:53,110 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:56:53,110 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:53,110 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:56:53,110 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:56:53,110 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:53,110 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:53,111 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:53,111 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:53,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:53,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:53,111 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:53,111 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:56:53,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:53,111 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:53,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:56:53,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:53,112 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:56:53,112 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:56:53,112 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:56:53,112 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:56:53,112 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:53,112 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:56:53,112 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:56:53,112 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:56:53,112 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:53,112 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. started... 2023-03-21 11:56:53,112 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. as already flushing 2023-03-21 11:56:53,112 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:53,112 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:56:53,112 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:56:53,112 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:53,112 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:56:53,112 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:56:53,113 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:53,113 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:53,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:53,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:53,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:53,113 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:53,113 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:56:53,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:53,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:53,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:53,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:53,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:53,114 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:56:53,114 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:56:53,114 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@46ecac08[Count = 0] remaining members to acquire global barrier 2023-03-21 11:56:53,114 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:53,115 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:53,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:53,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:53,115 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:53,115 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:56:53,115 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:56:53,115 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:56:53,115 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:56:53,116 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:53,116 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:56:53,116 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:53,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:53,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:53,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:53,116 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:56:53,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:53,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:53,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:53,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:53,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:53,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:53,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:53,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:56:53,118 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:56:53,118 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:56:53,118 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:56:53,118 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:56:53,118 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:56:53,118 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:56:53,118 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:53,118 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:53,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:53,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:53,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:53,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:53,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:53,119 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:53,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:53,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:53,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:53,119 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:53,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:53,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:53,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:53,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:53,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:53,120 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:53,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:53,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:53,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:53,121 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:53,121 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:53,121 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:53,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:53,121 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:56:53,121 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:56:53,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:53,121 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:53,122 DEBUG [Thread-1446] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:56:53,121 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:53,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:53,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:53,122 DEBUG [Thread-1446] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:56:53,122 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:53,122 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:53,122 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:53,122 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:53,122 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:53,222 DEBUG [Thread-1446] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:56:53,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:56:53,223 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:56:53,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:56:53,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:56:53,224 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:56:53,224 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:53,224 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:56:53,224 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:56:53,225 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:53,225 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:53,225 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:53,225 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:53,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:53,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:53,226 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:53,226 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:56:53,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:53,226 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:53,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:56:53,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:53,226 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:56:53,226 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:56:53,226 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:56:53,226 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:56:53,226 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:53,227 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:56:53,227 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:56:53,227 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:56:53,227 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:53,227 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. started... 2023-03-21 11:56:53,227 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. as already flushing 2023-03-21 11:56:53,227 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:53,227 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:56:53,227 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:56:53,227 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:53,227 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:56:53,227 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:56:53,228 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:53,228 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:53,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:53,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:53,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:53,228 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:53,228 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:56:53,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:53,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:53,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:53,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:53,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:53,229 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:56:53,229 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ae4cf58[Count = 0] remaining members to acquire global barrier 2023-03-21 11:56:53,229 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:56:53,229 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:53,230 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:53,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:53,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:53,230 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:56:53,230 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:56:53,230 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:56:53,230 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:53,230 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:56:53,231 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:56:53,231 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:53,231 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:53,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:53,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:53,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:53,231 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:56:53,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:53,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:53,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:53,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:53,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:53,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:53,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:53,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:56:53,232 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:56:53,232 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:56:53,233 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:56:53,233 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:56:53,233 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:56:53,233 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:56:53,234 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:53,234 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:53,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:53,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:53,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:53,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:53,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:53,234 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:53,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:53,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:53,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:53,234 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:53,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:53,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:53,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:53,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:53,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:53,235 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:53,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:53,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:53,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:53,237 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:53,237 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:53,237 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:53,237 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:53,237 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:53,237 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:53,237 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:53,237 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:56:53,237 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:56:53,237 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:56:53,237 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:53,237 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:53,237 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:53,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:53,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:53,237 DEBUG [Thread-1446] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:56:53,237 DEBUG [Thread-1446] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:56:53,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:53,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:53,259 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:53,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41730 deadline: 1679399873257, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:53,259 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:53,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41780 deadline: 1679399873258, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:53,259 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:53,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41768 deadline: 1679399873258, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:53,261 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:53,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41744 deadline: 1679399873260, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:53,337 DEBUG [Thread-1446] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:56:53,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:56:53,339 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:56:53,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:56:53,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:56:53,341 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:56:53,341 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:53,341 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:56:53,341 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:56:53,342 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:53,342 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:53,342 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:53,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:53,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:53,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:53,343 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:53,343 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:53,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:56:53,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:53,343 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:53,343 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:56:53,343 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:56:53,344 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:56:53,344 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:56:53,344 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:56:53,344 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:53,344 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:56:53,344 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:56:53,344 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:56:53,344 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:53,344 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. started... 2023-03-21 11:56:53,344 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. as already flushing 2023-03-21 11:56:53,344 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:53,344 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:56:53,344 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:56:53,344 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:53,344 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:56:53,344 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:56:53,345 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:53,345 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:53,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:53,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:53,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:53,346 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:53,346 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:56:53,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:53,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:53,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:53,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:53,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:53,347 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:56:53,347 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7bbdfa7c[Count = 0] remaining members to acquire global barrier 2023-03-21 11:56:53,347 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:56:53,347 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:53,348 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:53,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:53,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:53,348 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:56:53,348 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:56:53,348 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:53,348 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:56:53,348 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:56:53,349 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:56:53,349 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:53,349 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:53,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:53,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:53,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:53,349 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:56:53,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:53,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:53,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:53,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:53,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:53,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:53,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:53,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:56:53,351 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:56:53,351 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:56:53,351 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:56:53,351 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:56:53,351 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:56:53,351 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:56:53,351 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:53,351 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:53,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:53,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:53,351 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:53,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:53,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:53,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:53,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:53,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:53,352 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:53,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:53,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:53,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:53,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:53,353 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:53,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:53,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:53,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:53,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:53,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:53,354 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:53,354 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:53,354 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:53,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:53,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:53,354 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:53,354 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:56:53,355 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:56:53,354 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:53,355 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:56:53,354 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:53,355 DEBUG [Thread-1446] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:56:53,354 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:53,355 DEBUG [Thread-1446] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:56:53,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:53,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:53,355 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:53,355 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:53,355 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:53,455 DEBUG [Thread-1446] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:56:53,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:56:53,457 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:56:53,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:56:53,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:56:53,458 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:56:53,458 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:53,458 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:56:53,458 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:56:53,458 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:53,458 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:53,459 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:53,459 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:53,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:53,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:53,459 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:53,459 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:56:53,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:53,459 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:53,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:56:53,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:53,460 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:56:53,460 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:56:53,460 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:56:53,460 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:56:53,460 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:53,460 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:56:53,460 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:56:53,461 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:56:53,461 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:53,461 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. started... 2023-03-21 11:56:53,461 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. as already flushing 2023-03-21 11:56:53,461 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:53,461 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:56:53,461 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:56:53,461 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:53,461 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:56:53,461 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:56:53,462 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:53,462 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:53,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:53,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:53,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:53,462 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:53,462 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:56:53,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:53,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:53,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:53,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:53,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:53,463 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:56:53,463 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:56:53,463 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1bb1ea5b[Count = 0] remaining members to acquire global barrier 2023-03-21 11:56:53,463 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:53,463 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:53,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:53,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:53,464 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:56:53,464 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:53,464 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:56:53,464 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:56:53,464 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:56:53,464 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:53,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:53,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:53,465 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:56:53,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:53,465 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:53,465 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:56:53,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:53,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:53,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:53,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:53,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:53,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:53,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:53,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:56:53,466 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:56:53,466 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:56:53,466 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:56:53,467 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:56:53,467 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:56:53,467 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:56:53,467 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:53,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:53,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:53,467 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:53,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:53,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:53,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:53,468 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:53,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:53,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:53,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:53,468 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:53,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:53,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:53,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:53,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:53,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:53,469 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:53,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:53,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:53,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:53,470 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:53,470 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:53,470 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:53,470 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:53,470 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:53,470 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:53,470 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:56:53,471 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:56:53,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:53,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:53,470 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:53,470 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:53,471 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:53,471 DEBUG [Thread-1446] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:56:53,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:53,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:53,471 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:53,471 DEBUG [Thread-1446] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:56:53,477 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=247 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/B/556f41fecfaa4409b53d38e3693ffef4 2023-03-21 11:56:53,485 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=247 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/C/49532163ebb5459c885721be3443a74d 2023-03-21 11:56:53,488 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/A/a88afc7dc59744b2a8975118df873fed as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/a88afc7dc59744b2a8975118df873fed 2023-03-21 11:56:53,491 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/a88afc7dc59744b2a8975118df873fed, entries=200, sequenceid=247, filesize=14.1 K 2023-03-21 11:56:53,492 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/B/556f41fecfaa4409b53d38e3693ffef4 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/556f41fecfaa4409b53d38e3693ffef4 2023-03-21 11:56:53,495 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/556f41fecfaa4409b53d38e3693ffef4, entries=150, sequenceid=247, filesize=11.8 K 2023-03-21 11:56:53,496 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/C/49532163ebb5459c885721be3443a74d as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/49532163ebb5459c885721be3443a74d 2023-03-21 11:56:53,499 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/49532163ebb5459c885721be3443a74d, entries=150, sequenceid=247, filesize=11.8 K 2023-03-21 11:56:53,499 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~360 KB/368640, currentSize=53.67 KB/54960 for 886ee010948b04e81bf5ede3c083ff8b in 860ms, sequenceid=247, compaction requested=true 2023-03-21 11:56:53,499 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 886ee010948b04e81bf5ede3c083ff8b: 2023-03-21 11:56:53,499 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-21 11:56:53,499 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-21 11:56:53,499 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-21 11:56:53,499 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-21 11:56:53,500 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-21 11:56:53,500 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 51012 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-21 11:56:53,500 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48622 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-21 11:56:53,501 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1912): 886ee010948b04e81bf5ede3c083ff8b/A is initiating minor compaction (all files) 2023-03-21 11:56:53,501 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1912): 886ee010948b04e81bf5ede3c083ff8b/B is initiating minor compaction (all files) 2023-03-21 11:56:53,501 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 886ee010948b04e81bf5ede3c083ff8b/A in TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:53,501 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 886ee010948b04e81bf5ede3c083ff8b/B in TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:53,501 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/aec612a2b8c943feb6f8f623b01737d1, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/cb558ddc0c4c43e7bd5a417970368420, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/18b39d2506944108a539b603f6ecdc84, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/a88afc7dc59744b2a8975118df873fed] into tmpdir=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp, totalSize=49.8 K 2023-03-21 11:56:53,501 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/53d307cbe7f1425d8bc096bf778af22e, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/12fea09f3a5f4291902c1fbc5262beda, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/5506afddf73447acaa87276aaf0cb65e, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/556f41fecfaa4409b53d38e3693ffef4] into tmpdir=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp, totalSize=47.5 K 2023-03-21 11:56:53,501 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting aec612a2b8c943feb6f8f623b01737d1, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=191, earliestPutTs=1679399810003 2023-03-21 11:56:53,501 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 53d307cbe7f1425d8bc096bf778af22e, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=191, earliestPutTs=1679399810003 2023-03-21 11:56:53,502 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting cb558ddc0c4c43e7bd5a417970368420, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=210, earliestPutTs=1679399810351 2023-03-21 11:56:53,502 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 12fea09f3a5f4291902c1fbc5262beda, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=210, earliestPutTs=1679399810351 2023-03-21 11:56:53,502 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 18b39d2506944108a539b603f6ecdc84, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=226, earliestPutTs=1679399811704 2023-03-21 11:56:53,502 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 5506afddf73447acaa87276aaf0cb65e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=226, earliestPutTs=1679399811704 2023-03-21 11:56:53,502 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting a88afc7dc59744b2a8975118df873fed, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=247, earliestPutTs=1679399812331 2023-03-21 11:56:53,502 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 556f41fecfaa4409b53d38e3693ffef4, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=247, earliestPutTs=1679399812331 2023-03-21 11:56:53,509 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] throttle.PressureAwareThroughputController(145): 886ee010948b04e81bf5ede3c083ff8b#B#compaction#226 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-21 11:56:53,509 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] throttle.PressureAwareThroughputController(145): 886ee010948b04e81bf5ede3c083ff8b#A#compaction#225 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-21 11:56:53,518 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/A/4a6359f049cd4b6c8318f659247cb349 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/4a6359f049cd4b6c8318f659247cb349 2023-03-21 11:56:53,522 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 886ee010948b04e81bf5ede3c083ff8b/A of 886ee010948b04e81bf5ede3c083ff8b into 4a6359f049cd4b6c8318f659247cb349(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-21 11:56:53,522 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 886ee010948b04e81bf5ede3c083ff8b: 2023-03-21 11:56:53,522 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b., storeName=886ee010948b04e81bf5ede3c083ff8b/A, priority=12, startTime=1679399813499; duration=0sec 2023-03-21 11:56:53,522 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-21 11:56:53,522 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-21 11:56:53,523 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48622 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-21 11:56:53,523 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1912): 886ee010948b04e81bf5ede3c083ff8b/C is initiating minor compaction (all files) 2023-03-21 11:56:53,523 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 886ee010948b04e81bf5ede3c083ff8b/C in TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:53,523 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/1838cea787a5435ea08239055ab37b20, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/7ef9fe3da1e64dd3a63be4130e70ef27, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/2283fe3ccae44867ae8b2519e280987e, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/49532163ebb5459c885721be3443a74d] into tmpdir=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp, totalSize=47.5 K 2023-03-21 11:56:53,524 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 1838cea787a5435ea08239055ab37b20, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=191, earliestPutTs=1679399810003 2023-03-21 11:56:53,524 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 7ef9fe3da1e64dd3a63be4130e70ef27, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=210, earliestPutTs=1679399810351 2023-03-21 11:56:53,524 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 2283fe3ccae44867ae8b2519e280987e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=226, earliestPutTs=1679399811704 2023-03-21 11:56:53,524 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 49532163ebb5459c885721be3443a74d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=247, earliestPutTs=1679399812331 2023-03-21 11:56:53,531 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] throttle.PressureAwareThroughputController(145): 886ee010948b04e81bf5ede3c083ff8b#C#compaction#227 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-21 11:56:53,571 DEBUG [Thread-1446] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:56:53,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:56:53,573 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:56:53,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:56:53,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:56:53,574 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:56:53,574 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:53,574 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:56:53,574 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:56:53,574 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:53,574 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:53,575 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:53,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:53,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:53,575 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:53,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:53,575 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:53,575 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:56:53,575 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:53,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:56:53,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:53,576 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:56:53,576 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:56:53,576 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:56:53,576 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:56:53,576 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:53,576 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:56:53,576 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:56:53,576 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:56:53,576 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:53,576 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. started... 2023-03-21 11:56:53,576 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 886ee010948b04e81bf5ede3c083ff8b 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-21 11:56:53,577 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 886ee010948b04e81bf5ede3c083ff8b, store=A 2023-03-21 11:56:53,577 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:56:53,577 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 886ee010948b04e81bf5ede3c083ff8b, store=B 2023-03-21 11:56:53,577 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:56:53,577 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 886ee010948b04e81bf5ede3c083ff8b, store=C 2023-03-21 11:56:53,577 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:56:53,584 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=259 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/A/8b68dfbaf7314b119eb7eb7ee1b87da4 2023-03-21 11:56:53,613 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=259 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/B/81fe49d653d0495c8c62d4419197eef6 2023-03-21 11:56:53,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(9158): Flush requested on 886ee010948b04e81bf5ede3c083ff8b 2023-03-21 11:56:53,762 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. as already flushing 2023-03-21 11:56:53,778 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:53,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41730 deadline: 1679399873774, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:53,778 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:53,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41744 deadline: 1679399873775, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:53,779 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1679399813771, max timestamp=1679399813777Num uniques 859; 2023-03-21 11:56:53,779 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1679399813771, max timestamp=1679399813777Num uniques 859; 2023-03-21 11:56:53,779 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1679399813771, max timestamp=1679399813777Num uniques 859; 2023-03-21 11:56:53,779 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-21 11:56:53,779 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-21 11:56:53,779 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-21 11:56:53,817 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:53,817 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:53,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41768 deadline: 1679399873815, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:53,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41780 deadline: 1679399873815, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:53,880 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:53,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41730 deadline: 1679399873879, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:53,881 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:53,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41744 deadline: 1679399873879, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:53,916 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/B/7cf63990500243baa09700927d3c0e40 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/7cf63990500243baa09700927d3c0e40 2023-03-21 11:56:53,921 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 886ee010948b04e81bf5ede3c083ff8b/B of 886ee010948b04e81bf5ede3c083ff8b into 7cf63990500243baa09700927d3c0e40(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-21 11:56:53,921 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 886ee010948b04e81bf5ede3c083ff8b: 2023-03-21 11:56:53,921 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b., storeName=886ee010948b04e81bf5ede3c083ff8b/B, priority=12, startTime=1679399813499; duration=0sec 2023-03-21 11:56:53,921 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-21 11:56:53,921 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:53,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41768 deadline: 1679399873918, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:53,921 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:53,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41780 deadline: 1679399873918, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:53,956 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/C/c57a4dc71aaf4f67aabfc9bb90c488c6 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/c57a4dc71aaf4f67aabfc9bb90c488c6 2023-03-21 11:56:53,961 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 886ee010948b04e81bf5ede3c083ff8b/C of 886ee010948b04e81bf5ede3c083ff8b into c57a4dc71aaf4f67aabfc9bb90c488c6(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-21 11:56:53,961 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 886ee010948b04e81bf5ede3c083ff8b: 2023-03-21 11:56:53,961 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b., storeName=886ee010948b04e81bf5ede3c083ff8b/C, priority=12, startTime=1679399813500; duration=0sec 2023-03-21 11:56:53,961 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-21 11:56:54,030 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=259 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/C/ca53b302ab324123804b4decf61921b7 2023-03-21 11:56:54,034 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/A/8b68dfbaf7314b119eb7eb7ee1b87da4 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/8b68dfbaf7314b119eb7eb7ee1b87da4 2023-03-21 11:56:54,038 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/8b68dfbaf7314b119eb7eb7ee1b87da4, entries=150, sequenceid=259, filesize=11.8 K 2023-03-21 11:56:54,039 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/B/81fe49d653d0495c8c62d4419197eef6 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/81fe49d653d0495c8c62d4419197eef6 2023-03-21 11:56:54,042 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/81fe49d653d0495c8c62d4419197eef6, entries=150, sequenceid=259, filesize=11.8 K 2023-03-21 11:56:54,043 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/C/ca53b302ab324123804b4decf61921b7 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/ca53b302ab324123804b4decf61921b7 2023-03-21 11:56:54,052 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/ca53b302ab324123804b4decf61921b7, entries=150, sequenceid=259, filesize=11.8 K 2023-03-21 11:56:54,053 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 886ee010948b04e81bf5ede3c083ff8b in 477ms, sequenceid=259, compaction requested=false 2023-03-21 11:56:54,053 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 886ee010948b04e81bf5ede3c083ff8b: 2023-03-21 11:56:54,053 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:54,053 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:56:54,053 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:56:54,053 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:54,058 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:56:54,058 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:56:54,059 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:54,059 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:54,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:54,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:54,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:54,059 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:54,059 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:56:54,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:54,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:54,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:54,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:54,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:54,061 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:56:54,061 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@389e89c9[Count = 0] remaining members to acquire global barrier 2023-03-21 11:56:54,061 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:56:54,061 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:54,062 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:54,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:54,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:54,062 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:56:54,062 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:56:54,062 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:56:54,062 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:54,062 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:56:54,063 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:54,063 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:56:54,063 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:54,063 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:56:54,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:54,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:54,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:54,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:54,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:54,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:54,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:54,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:54,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:54,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:54,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:56:54,067 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:56:54,067 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:56:54,067 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:56:54,067 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:56:54,067 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:56:54,067 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:56:54,069 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:54,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:54,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:54,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:54,069 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:54,069 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:54,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:54,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:54,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:54,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:54,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:54,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:54,070 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:54,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:54,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:54,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:54,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:54,071 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:54,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:54,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:54,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:54,073 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:54,073 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:54,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:54,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:54,073 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:54,073 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:56:54,074 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:56:54,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:54,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:54,074 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:54,074 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:54,074 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:54,074 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:54,074 DEBUG [Thread-1446] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:56:54,074 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:54,074 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:54,074 DEBUG [Thread-1446] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:56:54,074 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:54,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(9158): Flush requested on 886ee010948b04e81bf5ede3c083ff8b 2023-03-21 11:56:54,084 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 886ee010948b04e81bf5ede3c083ff8b 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-03-21 11:56:54,084 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 886ee010948b04e81bf5ede3c083ff8b, store=A 2023-03-21 11:56:54,085 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-21 11:56:54,085 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 886ee010948b04e81bf5ede3c083ff8b, store=B 2023-03-21 11:56:54,085 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-21 11:56:54,085 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 886ee010948b04e81bf5ede3c083ff8b, store=C 2023-03-21 11:56:54,085 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-21 11:56:54,098 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:54,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41744 deadline: 1679399874096, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:54,102 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:54,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41730 deadline: 1679399874099, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:54,116 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=298 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/A/f19a6b74f5d548e3acc85ada8affe5de 2023-03-21 11:56:54,124 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:54,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41768 deadline: 1679399874122, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:54,125 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:54,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41780 deadline: 1679399874123, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:54,169 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=298 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/B/7e8e573d142943a1b76db7e9a62ee270 2023-03-21 11:56:54,174 DEBUG [Thread-1446] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:56:54,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:56:54,176 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:56:54,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:56:54,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:56:54,189 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:56:54,189 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:54,189 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:56:54,189 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:56:54,189 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:54,189 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:54,190 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:54,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:54,190 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:54,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:54,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:54,191 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:54,191 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:54,191 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:56:54,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:56:54,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:54,191 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:56:54,198 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:56:54,198 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:56:54,198 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:56:54,198 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:54,198 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:56:54,199 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:56:54,199 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:56:54,199 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:54,199 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. started... 2023-03-21 11:56:54,199 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. as already flushing 2023-03-21 11:56:54,199 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:54,199 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:56:54,199 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:56:54,199 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:54,200 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:54,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41744 deadline: 1679399874200, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:54,206 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:54,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41730 deadline: 1679399874204, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:54,210 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:56:54,210 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:56:54,211 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:54,211 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:54,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:54,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:54,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:54,212 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:54,212 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:56:54,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:54,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:54,213 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=298 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/C/1cabcceb8eba484095605a4db3b5ac7f 2023-03-21 11:56:54,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:54,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:54,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:54,215 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:56:54,215 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@44fc0f8d[Count = 0] remaining members to acquire global barrier 2023-03-21 11:56:54,215 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:56:54,215 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:54,215 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:54,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:54,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:54,216 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:56:54,216 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:56:54,216 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:56:54,216 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:54,216 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:56:54,217 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:56:54,217 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:54,217 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:56:54,219 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:54,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:54,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:54,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:54,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:54,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:54,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:54,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:54,220 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/A/f19a6b74f5d548e3acc85ada8affe5de as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/f19a6b74f5d548e3acc85ada8affe5de 2023-03-21 11:56:54,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:54,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:54,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:54,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:56:54,223 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:56:54,223 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:56:54,223 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:56:54,223 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:56:54,223 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:56:54,223 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:56:54,225 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/f19a6b74f5d548e3acc85ada8affe5de, entries=250, sequenceid=298, filesize=16.7 K 2023-03-21 11:56:54,227 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/B/7e8e573d142943a1b76db7e9a62ee270 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/7e8e573d142943a1b76db7e9a62ee270 2023-03-21 11:56:54,230 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/7e8e573d142943a1b76db7e9a62ee270, entries=150, sequenceid=298, filesize=11.9 K 2023-03-21 11:56:54,231 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/C/1cabcceb8eba484095605a4db3b5ac7f as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/1cabcceb8eba484095605a4db3b5ac7f 2023-03-21 11:56:54,235 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:54,235 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:54,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:54,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:54,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:54,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:54,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:54,235 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:54,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:54,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:54,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:54,238 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:54,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:54,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:54,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:54,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:54,240 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:54,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:54,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:54,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:54,242 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/1cabcceb8eba484095605a4db3b5ac7f, entries=150, sequenceid=298, filesize=11.9 K 2023-03-21 11:56:54,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:54,243 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~400.17 KB/409776, currentSize=40.25 KB/41220 for 886ee010948b04e81bf5ede3c083ff8b in 159ms, sequenceid=298, compaction requested=true 2023-03-21 11:56:54,243 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 886ee010948b04e81bf5ede3c083ff8b: 2023-03-21 11:56:54,243 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-21 11:56:54,243 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-21 11:56:54,243 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-21 11:56:54,243 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-21 11:56:54,243 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-21 11:56:54,244 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 41802 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-21 11:56:54,244 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1912): 886ee010948b04e81bf5ede3c083ff8b/A is initiating minor compaction (all files) 2023-03-21 11:56:54,244 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36922 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-21 11:56:54,244 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 886ee010948b04e81bf5ede3c083ff8b/A in TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:54,244 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1912): 886ee010948b04e81bf5ede3c083ff8b/B is initiating minor compaction (all files) 2023-03-21 11:56:54,244 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/4a6359f049cd4b6c8318f659247cb349, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/8b68dfbaf7314b119eb7eb7ee1b87da4, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/f19a6b74f5d548e3acc85ada8affe5de] into tmpdir=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp, totalSize=40.8 K 2023-03-21 11:56:54,244 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 886ee010948b04e81bf5ede3c083ff8b/B in TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:54,244 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/7cf63990500243baa09700927d3c0e40, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/81fe49d653d0495c8c62d4419197eef6, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/7e8e573d142943a1b76db7e9a62ee270] into tmpdir=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp, totalSize=36.1 K 2023-03-21 11:56:54,244 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 4a6359f049cd4b6c8318f659247cb349, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=247, earliestPutTs=1679399812331 2023-03-21 11:56:54,244 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 7cf63990500243baa09700927d3c0e40, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=247, earliestPutTs=1679399812331 2023-03-21 11:56:54,245 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 8b68dfbaf7314b119eb7eb7ee1b87da4, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=259, earliestPutTs=1679399812647 2023-03-21 11:56:54,245 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 81fe49d653d0495c8c62d4419197eef6, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=259, earliestPutTs=1679399812647 2023-03-21 11:56:54,245 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting f19a6b74f5d548e3acc85ada8affe5de, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=298, earliestPutTs=1679399813800 2023-03-21 11:56:54,245 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 7e8e573d142943a1b76db7e9a62ee270, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=298, earliestPutTs=1679399813800 2023-03-21 11:56:54,256 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:54,256 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:54,256 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:54,256 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:56:54,256 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:56:54,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:54,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:54,256 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:54,256 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:56:54,256 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:54,256 DEBUG [Thread-1446] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:56:54,256 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:54,256 DEBUG [Thread-1446] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:56:54,256 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:54,257 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:54,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:54,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:54,257 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:54,257 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:54,271 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] throttle.PressureAwareThroughputController(145): 886ee010948b04e81bf5ede3c083ff8b#A#compaction#234 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-21 11:56:54,275 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] throttle.PressureAwareThroughputController(145): 886ee010948b04e81bf5ede3c083ff8b#B#compaction#235 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-21 11:56:54,327 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/B/1a78dcc7492b4daf89506206e57a5d20 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/1a78dcc7492b4daf89506206e57a5d20 2023-03-21 11:56:54,331 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/A/5f32f0c676994684acb666ac96ab9d71 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/5f32f0c676994684acb666ac96ab9d71 2023-03-21 11:56:54,338 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 886ee010948b04e81bf5ede3c083ff8b/A of 886ee010948b04e81bf5ede3c083ff8b into 5f32f0c676994684acb666ac96ab9d71(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-21 11:56:54,338 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 886ee010948b04e81bf5ede3c083ff8b: 2023-03-21 11:56:54,338 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b., storeName=886ee010948b04e81bf5ede3c083ff8b/A, priority=13, startTime=1679399814243; duration=0sec 2023-03-21 11:56:54,338 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-21 11:56:54,338 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-21 11:56:54,339 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36922 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-21 11:56:54,339 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1912): 886ee010948b04e81bf5ede3c083ff8b/C is initiating minor compaction (all files) 2023-03-21 11:56:54,339 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 886ee010948b04e81bf5ede3c083ff8b/C in TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:54,339 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/c57a4dc71aaf4f67aabfc9bb90c488c6, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/ca53b302ab324123804b4decf61921b7, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/1cabcceb8eba484095605a4db3b5ac7f] into tmpdir=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp, totalSize=36.1 K 2023-03-21 11:56:54,340 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting c57a4dc71aaf4f67aabfc9bb90c488c6, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=247, earliestPutTs=1679399812331 2023-03-21 11:56:54,340 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting ca53b302ab324123804b4decf61921b7, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=259, earliestPutTs=1679399812647 2023-03-21 11:56:54,340 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 886ee010948b04e81bf5ede3c083ff8b/B of 886ee010948b04e81bf5ede3c083ff8b into 1a78dcc7492b4daf89506206e57a5d20(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-21 11:56:54,340 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 886ee010948b04e81bf5ede3c083ff8b: 2023-03-21 11:56:54,340 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b., storeName=886ee010948b04e81bf5ede3c083ff8b/B, priority=13, startTime=1679399814243; duration=0sec 2023-03-21 11:56:54,340 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 1cabcceb8eba484095605a4db3b5ac7f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=298, earliestPutTs=1679399813800 2023-03-21 11:56:54,340 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-21 11:56:54,346 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] throttle.PressureAwareThroughputController(145): 886ee010948b04e81bf5ede3c083ff8b#C#compaction#236 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-21 11:56:54,356 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/C/b798803d08ba461c910d60a95eb91e95 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/b798803d08ba461c910d60a95eb91e95 2023-03-21 11:56:54,357 DEBUG [Thread-1446] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:56:54,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:56:54,359 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:56:54,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:56:54,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:56:54,360 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:56:54,360 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:54,360 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:56:54,360 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:56:54,361 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:54,361 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:54,361 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:54,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:54,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:54,362 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:54,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:54,362 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:54,362 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:56:54,362 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:54,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:56:54,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:54,362 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:56:54,363 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:56:54,363 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:56:54,363 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:56:54,363 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:54,363 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:56:54,365 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 886ee010948b04e81bf5ede3c083ff8b/C of 886ee010948b04e81bf5ede3c083ff8b into b798803d08ba461c910d60a95eb91e95(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-21 11:56:54,365 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 886ee010948b04e81bf5ede3c083ff8b: 2023-03-21 11:56:54,365 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b., storeName=886ee010948b04e81bf5ede3c083ff8b/C, priority=13, startTime=1679399814243; duration=0sec 2023-03-21 11:56:54,365 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-21 11:56:54,370 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:56:54,370 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:54,370 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:56:54,370 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. started... 2023-03-21 11:56:54,370 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 886ee010948b04e81bf5ede3c083ff8b 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-03-21 11:56:54,370 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 886ee010948b04e81bf5ede3c083ff8b, store=A 2023-03-21 11:56:54,370 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:56:54,370 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 886ee010948b04e81bf5ede3c083ff8b, store=B 2023-03-21 11:56:54,371 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:56:54,371 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 886ee010948b04e81bf5ede3c083ff8b, store=C 2023-03-21 11:56:54,371 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:56:54,396 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=310 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/A/1b5080d7ef444b328fa937fa566802a5 2023-03-21 11:56:54,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(9158): Flush requested on 886ee010948b04e81bf5ede3c083ff8b 2023-03-21 11:56:54,405 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. as already flushing 2023-03-21 11:56:54,433 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679399814424, max timestamp=1679399814432Num uniques 859; 2023-03-21 11:56:54,434 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-21 11:56:54,434 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679399814424, max timestamp=1679399814432Num uniques 859; 2023-03-21 11:56:54,434 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-21 11:56:54,434 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679399814424, max timestamp=1679399814432Num uniques 859; 2023-03-21 11:56:54,434 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-21 11:56:54,454 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1679399814442, max timestamp=1679399814452Num uniques 1018; 2023-03-21 11:56:54,454 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-03-21 11:56:54,454 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1679399814442, max timestamp=1679399814452Num uniques 1018; 2023-03-21 11:56:54,454 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1679399814442, max timestamp=1679399814452Num uniques 1018; 2023-03-21 11:56:54,454 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-03-21 11:56:54,455 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-03-21 11:56:54,470 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:54,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41744 deadline: 1679399874468, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:54,471 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:54,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41780 deadline: 1679399874469, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:54,472 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:54,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41768 deadline: 1679399874470, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:54,472 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:54,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41730 deadline: 1679399874470, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:54,490 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:54,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41826 deadline: 1679399874489, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:54,492 DEBUG [Thread-1440] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4128 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b., hostname=jenkins-hbase20.apache.org,34369,1679399732030, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-21 11:56:54,574 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:54,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41744 deadline: 1679399874571, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:54,574 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:54,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41768 deadline: 1679399874572, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:54,575 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:54,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41730 deadline: 1679399874573, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:54,575 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:54,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41780 deadline: 1679399874573, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:54,776 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:54,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41768 deadline: 1679399874775, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:54,777 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:54,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41744 deadline: 1679399874776, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:54,777 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:54,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41730 deadline: 1679399874776, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:54,780 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:54,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41780 deadline: 1679399874777, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:54,821 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=310 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/B/2933409c7e814eb687ac0f3336f0d780 2023-03-21 11:56:54,838 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=310 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/C/d96a44630578478fbdc148c147373a97 2023-03-21 11:56:54,844 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/A/1b5080d7ef444b328fa937fa566802a5 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/1b5080d7ef444b328fa937fa566802a5 2023-03-21 11:56:54,851 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/1b5080d7ef444b328fa937fa566802a5, entries=150, sequenceid=310, filesize=11.9 K 2023-03-21 11:56:54,851 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/B/2933409c7e814eb687ac0f3336f0d780 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/2933409c7e814eb687ac0f3336f0d780 2023-03-21 11:56:54,857 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/2933409c7e814eb687ac0f3336f0d780, entries=150, sequenceid=310, filesize=11.9 K 2023-03-21 11:56:54,858 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/C/d96a44630578478fbdc148c147373a97 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/d96a44630578478fbdc148c147373a97 2023-03-21 11:56:54,863 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/d96a44630578478fbdc148c147373a97, entries=150, sequenceid=310, filesize=11.9 K 2023-03-21 11:56:54,864 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=147.60 KB/151140 for 886ee010948b04e81bf5ede3c083ff8b in 494ms, sequenceid=310, compaction requested=false 2023-03-21 11:56:54,864 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 886ee010948b04e81bf5ede3c083ff8b: 2023-03-21 11:56:54,864 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:54,864 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:56:54,864 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:56:54,864 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:54,864 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:56:54,864 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:56:54,865 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:54,865 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:54,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:54,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:54,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:54,865 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:54,865 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:56:54,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:54,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:54,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:54,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:54,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:54,866 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:56:54,866 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50067374[Count = 0] remaining members to acquire global barrier 2023-03-21 11:56:54,866 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:56:54,866 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:54,867 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:54,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:54,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:54,867 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:56:54,867 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:56:54,867 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:56:54,868 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:54,868 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:56:54,868 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:54,868 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:56:54,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:54,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:54,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:54,868 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:54,868 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:56:54,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:54,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:54,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:54,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:54,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:54,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:54,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:54,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:56:54,871 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:56:54,871 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:56:54,871 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:56:54,871 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:56:54,871 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:56:54,871 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:56:54,872 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:54,872 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:54,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:54,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:54,872 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:54,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:54,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:54,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:54,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:54,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:54,873 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:54,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:54,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:54,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:54,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:54,874 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:54,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:54,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:54,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:54,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:54,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:54,875 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:54,875 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:54,875 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:54,875 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:54,875 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:54,875 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:54,875 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:54,875 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:54,875 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:54,875 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:54,875 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:56:54,875 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:56:54,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:54,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:54,875 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:56:54,875 DEBUG [Thread-1446] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:56:54,876 DEBUG [Thread-1446] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:56:54,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:54,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:54,976 DEBUG [Thread-1446] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:56:54,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:56:54,977 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:56:54,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:56:54,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:56:54,978 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:56:54,978 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:54,978 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:56:54,978 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:56:54,979 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:54,979 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:54,979 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:54,979 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:54,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:54,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:54,980 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:54,980 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:56:54,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:54,980 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:54,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:56:54,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:54,980 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:56:54,981 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:56:54,981 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:56:54,981 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:56:54,981 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:54,981 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:56:54,981 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:56:54,981 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:56:54,981 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:54,981 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. started... 2023-03-21 11:56:54,982 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 886ee010948b04e81bf5ede3c083ff8b 3/3 column families, dataSize=147.60 KB heapSize=414.61 KB 2023-03-21 11:56:54,982 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 886ee010948b04e81bf5ede3c083ff8b, store=A 2023-03-21 11:56:54,982 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-21 11:56:54,982 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 886ee010948b04e81bf5ede3c083ff8b, store=B 2023-03-21 11:56:54,982 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-21 11:56:54,982 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 886ee010948b04e81bf5ede3c083ff8b, store=C 2023-03-21 11:56:54,982 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-21 11:56:55,007 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=364 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/A/a17b925816d2443f888acac99c38dcf8 2023-03-21 11:56:55,019 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=364 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/B/c4e09e3793bd455bab52bd48d7cdeafc 2023-03-21 11:56:55,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(9158): Flush requested on 886ee010948b04e81bf5ede3c083ff8b 2023-03-21 11:56:55,081 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. as already flushing 2023-03-21 11:56:55,086 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:55,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41730 deadline: 1679399875082, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:55,086 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:55,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41780 deadline: 1679399875082, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:55,087 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:55,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41744 deadline: 1679399875085, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:55,088 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:55,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41768 deadline: 1679399875086, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:55,189 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:55,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41730 deadline: 1679399875187, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:55,189 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:55,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41744 deadline: 1679399875187, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:55,189 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:55,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41768 deadline: 1679399875189, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:55,391 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:55,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41744 deadline: 1679399875390, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:55,392 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:55,392 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:55,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41768 deadline: 1679399875390, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:55,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41730 deadline: 1679399875390, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:55,429 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=364 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/C/14c19255426741d98459554ed87ffc70 2023-03-21 11:56:55,432 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/A/a17b925816d2443f888acac99c38dcf8 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/a17b925816d2443f888acac99c38dcf8 2023-03-21 11:56:55,438 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/a17b925816d2443f888acac99c38dcf8, entries=150, sequenceid=364, filesize=11.9 K 2023-03-21 11:56:55,438 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/B/c4e09e3793bd455bab52bd48d7cdeafc as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/c4e09e3793bd455bab52bd48d7cdeafc 2023-03-21 11:56:55,444 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/c4e09e3793bd455bab52bd48d7cdeafc, entries=150, sequenceid=364, filesize=11.9 K 2023-03-21 11:56:55,445 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/C/14c19255426741d98459554ed87ffc70 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/14c19255426741d98459554ed87ffc70 2023-03-21 11:56:55,450 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/14c19255426741d98459554ed87ffc70, entries=150, sequenceid=364, filesize=11.9 K 2023-03-21 11:56:55,451 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~414.56 KB/424512, currentSize=33.54 KB/34350 for 886ee010948b04e81bf5ede3c083ff8b in 469ms, sequenceid=364, compaction requested=true 2023-03-21 11:56:55,451 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 886ee010948b04e81bf5ede3c083ff8b: 2023-03-21 11:56:55,451 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:55,451 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:56:55,451 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:56:55,451 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:55,454 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:56:55,454 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:56:55,455 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:55,455 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:55,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:55,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:55,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:55,455 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:55,455 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:56:55,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:55,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:55,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:55,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:55,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:55,456 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:56:55,456 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4471b81d[Count = 0] remaining members to acquire global barrier 2023-03-21 11:56:55,456 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:56:55,456 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:55,457 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:55,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:55,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:55,457 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:56:55,457 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:56:55,457 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:56:55,457 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:55,457 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:56:55,458 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:55,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:55,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:55,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:55,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:55,458 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:56:55,458 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:55,458 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:56:55,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:55,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:55,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:55,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:55,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:55,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:55,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:56:55,460 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:56:55,460 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:56:55,460 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:56:55,460 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:56:55,460 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:56:55,460 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:56:55,461 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:55,461 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:55,461 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:55,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:55,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:55,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:55,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:55,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:55,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:55,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:55,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:55,463 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:55,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:55,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:55,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:55,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:55,467 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:55,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:55,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:55,474 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:55,474 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:55,474 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:55,474 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:55,474 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:55,474 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:55,474 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:55,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:55,475 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:56:55,475 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:55,474 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:56:55,475 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:56:55,475 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:55,475 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:55,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:55,475 DEBUG [Thread-1446] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:56:55,475 DEBUG [Thread-1446] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:56:55,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:55,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:55,575 DEBUG [Thread-1446] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:56:55,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:56:55,577 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:56:55,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:56:55,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:56:55,579 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:56:55,579 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:55,579 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:56:55,579 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:56:55,579 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:55,579 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:55,580 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:55,580 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:55,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:55,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:55,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:55,580 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:55,580 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:56:55,581 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:55,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:56:55,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:55,581 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:56:55,581 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:56:55,581 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:56:55,581 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:56:55,581 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:55,581 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:56:55,581 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:56:55,581 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:56:55,581 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:55,581 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. started... 2023-03-21 11:56:55,582 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 886ee010948b04e81bf5ede3c083ff8b 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-03-21 11:56:55,582 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 886ee010948b04e81bf5ede3c083ff8b, store=A 2023-03-21 11:56:55,582 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:56:55,582 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 886ee010948b04e81bf5ede3c083ff8b, store=B 2023-03-21 11:56:55,582 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:56:55,582 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 886ee010948b04e81bf5ede3c083ff8b, store=C 2023-03-21 11:56:55,582 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:56:55,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(9158): Flush requested on 886ee010948b04e81bf5ede3c083ff8b 2023-03-21 11:56:55,589 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. as already flushing 2023-03-21 11:56:55,625 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679399815597, max timestamp=1679399815623Num uniques 859; 2023-03-21 11:56:55,625 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679399815597, max timestamp=1679399815623Num uniques 859; 2023-03-21 11:56:55,633 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679399815597, max timestamp=1679399815623Num uniques 859; 2023-03-21 11:56:55,633 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-21 11:56:55,633 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-21 11:56:55,633 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-21 11:56:55,684 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1679399815674, max timestamp=1679399815682Num uniques 1018; 2023-03-21 11:56:55,684 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-03-21 11:56:55,685 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1679399815674, max timestamp=1679399815682Num uniques 1018; 2023-03-21 11:56:55,685 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-03-21 11:56:55,685 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1679399815674, max timestamp=1679399815682Num uniques 1018; 2023-03-21 11:56:55,685 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-03-21 11:56:55,710 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1679399815695, max timestamp=1679399815708Num uniques 1027; 2023-03-21 11:56:55,710 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=12682, suffix heap size=134776,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=277 2023-03-21 11:56:55,711 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1679399815695, max timestamp=1679399815708Num uniques 1027; 2023-03-21 11:56:55,711 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1679399815695, max timestamp=1679399815708Num uniques 1027; 2023-03-21 11:56:55,711 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=12682, suffix heap size=134776,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=277 2023-03-21 11:56:55,711 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=12682, suffix heap size=134776,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=277 2023-03-21 11:56:55,726 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:55,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41744 deadline: 1679399875722, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:55,726 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:55,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41730 deadline: 1679399875723, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:55,728 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1679399815720, max timestamp=1679399815725Num uniques 1136; 2023-03-21 11:56:55,728 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=8516, suffix heap size=146112,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=186 2023-03-21 11:56:55,728 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1679399815720, max timestamp=1679399815725Num uniques 1136; 2023-03-21 11:56:55,728 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1679399815720, max timestamp=1679399815725Num uniques 1136; 2023-03-21 11:56:55,728 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=8516, suffix heap size=146112,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=186 2023-03-21 11:56:55,728 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=8516, suffix heap size=146112,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=186 2023-03-21 11:56:55,755 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1679399815744, max timestamp=1679399815753Num uniques 1045; 2023-03-21 11:56:55,755 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=11220, suffix heap size=136648,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=245 2023-03-21 11:56:55,755 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1679399815744, max timestamp=1679399815753Num uniques 1045; 2023-03-21 11:56:55,755 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=11220, suffix heap size=136648,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=245 2023-03-21 11:56:55,756 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1679399815744, max timestamp=1679399815753Num uniques 1045; 2023-03-21 11:56:55,756 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=11220, suffix heap size=136648,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=245 2023-03-21 11:56:55,782 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:55,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41780 deadline: 1679399875780, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:55,783 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:55,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41768 deadline: 1679399875780, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:55,829 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:55,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41744 deadline: 1679399875827, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:55,830 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:55,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41730 deadline: 1679399875828, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:55,884 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:55,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41780 deadline: 1679399875883, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:55,885 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:55,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41768 deadline: 1679399875883, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:55,989 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=372 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/A/1ae9ddb1370b4d26bf263ec31a0eb180 2023-03-21 11:56:56,032 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:56,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41744 deadline: 1679399876030, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:56,032 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:56,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41730 deadline: 1679399876031, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:56,087 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:56,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41780 deadline: 1679399876085, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:56,088 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:56,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41768 deadline: 1679399876086, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:56,335 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:56,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41730 deadline: 1679399876333, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:56,335 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:56,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41744 deadline: 1679399876334, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:56,391 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:56,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41768 deadline: 1679399876390, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:56,391 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:56,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41780 deadline: 1679399876390, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:56,397 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=372 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/B/aefe42cada67458d9522aacfe7037a42 2023-03-21 11:56:56,405 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=372 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/C/5865371a0afd4e39ace91c13292aaa5a 2023-03-21 11:56:56,408 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/A/1ae9ddb1370b4d26bf263ec31a0eb180 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/1ae9ddb1370b4d26bf263ec31a0eb180 2023-03-21 11:56:56,411 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/1ae9ddb1370b4d26bf263ec31a0eb180, entries=100, sequenceid=372, filesize=9.5 K 2023-03-21 11:56:56,412 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/B/aefe42cada67458d9522aacfe7037a42 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/aefe42cada67458d9522aacfe7037a42 2023-03-21 11:56:56,415 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/aefe42cada67458d9522aacfe7037a42, entries=100, sequenceid=372, filesize=9.5 K 2023-03-21 11:56:56,416 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/C/5865371a0afd4e39ace91c13292aaa5a as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/5865371a0afd4e39ace91c13292aaa5a 2023-03-21 11:56:56,420 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/5865371a0afd4e39ace91c13292aaa5a, entries=100, sequenceid=372, filesize=9.5 K 2023-03-21 11:56:56,421 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=147.60 KB/151140 for 886ee010948b04e81bf5ede3c083ff8b in 839ms, sequenceid=372, compaction requested=true 2023-03-21 11:56:56,421 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 886ee010948b04e81bf5ede3c083ff8b: 2023-03-21 11:56:56,421 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:56,422 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:56:56,422 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:56:56,422 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:56,422 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:56:56,422 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:56:56,423 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:56,423 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:56,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:56,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:56,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:56,423 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:56,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:56,423 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:56:56,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:56,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:56,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:56,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:56,424 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:56:56,424 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:56:56,424 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@8eda19b[Count = 0] remaining members to acquire global barrier 2023-03-21 11:56:56,424 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:56,425 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:56,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:56,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:56,425 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:56:56,425 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:56:56,425 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:56,425 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:56:56,425 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:56:56,426 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:56,426 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:56:56,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:56,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:56,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:56,426 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:56,426 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:56:56,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:56,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:56,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:56,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:56,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:56,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:56,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:56,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:56:56,428 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:56:56,428 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:56:56,428 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:56:56,428 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:56:56,428 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:56:56,428 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:56:56,429 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:56,429 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:56,429 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:56,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:56,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:56,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:56,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:56,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:56,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:56,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:56,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:56,430 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:56,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:56,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:56,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:56,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:56,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:56,431 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:56,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:56,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:56,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:56,433 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:56,433 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:56,433 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:56,433 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:56,433 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:56,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:56,433 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:56:56,433 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:56:56,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:56,433 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:56,433 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:56,433 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:56,433 DEBUG [Thread-1446] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:56:56,433 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:56,434 DEBUG [Thread-1446] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:56:56,434 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:56,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:56,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:56,534 DEBUG [Thread-1446] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:56:56,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:56:56,535 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:56:56,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:56:56,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:56:56,536 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:56:56,536 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:56,536 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:56:56,536 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:56:56,537 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:56,537 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:56,537 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:56,537 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:56,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:56,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:56,538 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:56,538 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:56:56,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:56,538 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:56,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:56:56,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:56,538 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:56:56,538 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:56:56,538 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:56:56,539 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:56:56,539 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:56,539 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:56:56,539 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:56:56,539 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:56:56,539 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:56,539 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. started... 2023-03-21 11:56:56,539 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 886ee010948b04e81bf5ede3c083ff8b 3/3 column families, dataSize=147.60 KB heapSize=416.77 KB 2023-03-21 11:56:56,539 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 886ee010948b04e81bf5ede3c083ff8b, store=A 2023-03-21 11:56:56,539 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-21 11:56:56,539 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 886ee010948b04e81bf5ede3c083ff8b, store=B 2023-03-21 11:56:56,539 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-21 11:56:56,539 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 886ee010948b04e81bf5ede3c083ff8b, store=C 2023-03-21 11:56:56,540 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-21 11:56:56,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(9158): Flush requested on 886ee010948b04e81bf5ede3c083ff8b 2023-03-21 11:56:56,838 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. as already flushing 2023-03-21 11:56:56,846 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:56,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41744 deadline: 1679399876844, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:56,846 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:56,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41730 deadline: 1679399876844, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:56,896 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:56,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41768 deadline: 1679399876895, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:56,898 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:56,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41780 deadline: 1679399876896, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:56,946 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=478 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/A/80d691407a124458bd61a335298416db 2023-03-21 11:56:56,948 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:56,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41744 deadline: 1679399876947, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:56,948 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:56,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41730 deadline: 1679399876947, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:57,150 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:57,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41744 deadline: 1679399877149, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:57,152 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:57,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41730 deadline: 1679399877150, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:57,355 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=478 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/B/09944657c93748f8a3cbbce23a1ddb8d 2023-03-21 11:56:57,453 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:57,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41744 deadline: 1679399877451, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:57,456 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:57,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41730 deadline: 1679399877455, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:57,764 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=478 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/C/3bbcf6077c8647cb91b582719621334f 2023-03-21 11:56:57,767 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/A/80d691407a124458bd61a335298416db as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/80d691407a124458bd61a335298416db 2023-03-21 11:56:57,770 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/80d691407a124458bd61a335298416db, entries=150, sequenceid=478, filesize=11.9 K 2023-03-21 11:56:57,771 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/B/09944657c93748f8a3cbbce23a1ddb8d as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/09944657c93748f8a3cbbce23a1ddb8d 2023-03-21 11:56:57,774 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/09944657c93748f8a3cbbce23a1ddb8d, entries=150, sequenceid=478, filesize=11.9 K 2023-03-21 11:56:57,775 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/C/3bbcf6077c8647cb91b582719621334f as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/3bbcf6077c8647cb91b582719621334f 2023-03-21 11:56:57,778 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/3bbcf6077c8647cb91b582719621334f, entries=150, sequenceid=478, filesize=11.9 K 2023-03-21 11:56:57,779 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~416.72 KB/426720, currentSize=33.54 KB/34350 for 886ee010948b04e81bf5ede3c083ff8b in 1240ms, sequenceid=478, compaction requested=true 2023-03-21 11:56:57,779 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 886ee010948b04e81bf5ede3c083ff8b: 2023-03-21 11:56:57,779 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:57,779 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:56:57,779 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:56:57,779 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:57,779 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:56:57,779 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:56:57,780 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:57,780 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:57,780 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:56:57,782 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:57,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:57,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:57,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:57,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:57,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:57,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:57,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:57,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:57,783 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:56:57,783 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2cc9476c[Count = 0] remaining members to acquire global barrier 2023-03-21 11:56:57,783 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:56:57,783 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:57,783 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:57,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:57,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:57,784 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:56:57,784 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:56:57,784 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:56:57,784 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:57,784 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:56:57,784 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:57,784 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:56:57,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:57,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:57,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:57,784 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:57,784 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:56:57,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:57,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:57,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:57,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:57,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:57,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:57,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:57,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:56:57,786 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:56:57,786 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:56:57,786 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:56:57,786 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:56:57,786 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:56:57,786 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:56:57,787 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:57,787 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:57,787 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:57,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:57,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:57,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:57,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:57,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:57,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:57,787 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:57,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:57,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:57,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:57,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:57,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:57,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:57,788 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:57,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:57,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:57,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:57,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:57,790 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:57,790 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:57,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:57,790 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:57,790 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:56:57,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:57,790 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:57,790 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:56:57,790 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:56:57,790 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:57,791 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:57,791 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:57,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:57,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:57,791 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:57,791 DEBUG [Thread-1446] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:56:57,791 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:57,791 DEBUG [Thread-1446] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:56:57,791 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:57,891 DEBUG [Thread-1446] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:56:57,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:56:57,892 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:56:57,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:56:57,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:56:57,893 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:56:57,893 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:57,894 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:56:57,894 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:56:57,894 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:57,894 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:57,895 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:57,895 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:57,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:57,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:57,895 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:57,895 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:56:57,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:57,896 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:57,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:56:57,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:57,896 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:56:57,897 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:56:57,897 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:56:57,897 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:56:57,897 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:57,897 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:56:57,897 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:56:57,897 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:56:57,897 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:57,897 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. started... 2023-03-21 11:56:57,897 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 886ee010948b04e81bf5ede3c083ff8b 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-03-21 11:56:57,897 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 886ee010948b04e81bf5ede3c083ff8b, store=A 2023-03-21 11:56:57,897 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:56:57,897 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 886ee010948b04e81bf5ede3c083ff8b, store=B 2023-03-21 11:56:57,897 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:56:57,897 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 886ee010948b04e81bf5ede3c083ff8b, store=C 2023-03-21 11:56:57,897 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:56:57,903 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=486 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/A/244c8cc84a734307aebcf39a362bedf3 2023-03-21 11:56:57,914 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=486 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/B/953cdf5eb2864dd0b7f130d8c0a41278 2023-03-21 11:56:57,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(9158): Flush requested on 886ee010948b04e81bf5ede3c083ff8b 2023-03-21 11:56:57,917 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. as already flushing 2023-03-21 11:56:57,946 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679399817929, max timestamp=1679399817944Num uniques 859; 2023-03-21 11:56:57,946 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679399817929, max timestamp=1679399817944Num uniques 859; 2023-03-21 11:56:57,946 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679399817929, max timestamp=1679399817944Num uniques 859; 2023-03-21 11:56:57,946 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-21 11:56:57,946 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-21 11:56:57,946 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-21 11:56:57,969 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:57,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41768 deadline: 1679399877967, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:57,971 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1679399817959, max timestamp=1679399817968Num uniques 1068; 2023-03-21 11:56:57,971 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1679399817963, max timestamp=1679399817968Num uniques 1068; 2023-03-21 11:56:57,971 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=12268, suffix heap size=139040,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=268 2023-03-21 11:56:57,971 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1679399817963, max timestamp=1679399817968Num uniques 1068; 2023-03-21 11:56:57,971 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-03-21 11:56:57,971 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-03-21 11:56:57,993 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1679399817976, max timestamp=1679399817992Num uniques 1127; 2023-03-21 11:56:57,993 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=10392, suffix heap size=145176,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=227 2023-03-21 11:56:57,994 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1679399817976, max timestamp=1679399817992Num uniques 1027; 2023-03-21 11:56:57,994 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-03-21 11:56:57,994 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1679399817976, max timestamp=1679399817992Num uniques 1027; 2023-03-21 11:56:57,994 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-03-21 11:56:58,018 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:58,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41744 deadline: 1679399878013, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:58,019 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8506, totalHeapSize=19584, min timestamp=1679399818002, max timestamp=1679399818016Num uniques 1086; 2023-03-21 11:56:58,019 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=49726, new segment data size=8506, suffix heap size=140912,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=186 2023-03-21 11:56:58,019 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8513, totalHeapSize=19584, min timestamp=1679399818002, max timestamp=1679399818016Num uniques 1086; 2023-03-21 11:56:58,019 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=49733, new segment data size=8513, suffix heap size=140912,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=186 2023-03-21 11:56:58,019 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8511, totalHeapSize=19584, min timestamp=1679399818002, max timestamp=1679399818016Num uniques 1086; 2023-03-21 11:56:58,019 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=49731, new segment data size=8511, suffix heap size=140912,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=186 2023-03-21 11:56:58,054 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1679399818043, max timestamp=1679399818052Num uniques 1045; 2023-03-21 11:56:58,055 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=11220, suffix heap size=136648,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=245 2023-03-21 11:56:58,055 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1679399818043, max timestamp=1679399818052Num uniques 1045; 2023-03-21 11:56:58,055 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=11220, suffix heap size=136648,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=245 2023-03-21 11:56:58,055 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1679399818043, max timestamp=1679399818052Num uniques 1045; 2023-03-21 11:56:58,055 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=11220, suffix heap size=136648,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=245 2023-03-21 11:56:58,088 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:58,089 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:58,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41730 deadline: 1679399878084, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:58,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 258 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41780 deadline: 1679399878085, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:58,091 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:58,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41768 deadline: 1679399878087, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:58,121 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:58,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41744 deadline: 1679399878120, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:58,192 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:58,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 260 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41780 deadline: 1679399878190, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:58,192 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:58,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41730 deadline: 1679399878190, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:58,195 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:58,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41768 deadline: 1679399878192, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:58,325 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:58,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41744 deadline: 1679399878323, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:58,331 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=486 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/C/e95a7ba93afe4ab8b0b99e1160522c89 2023-03-21 11:56:58,335 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/A/244c8cc84a734307aebcf39a362bedf3 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/244c8cc84a734307aebcf39a362bedf3 2023-03-21 11:56:58,338 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/244c8cc84a734307aebcf39a362bedf3, entries=150, sequenceid=486, filesize=11.9 K 2023-03-21 11:56:58,338 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/B/953cdf5eb2864dd0b7f130d8c0a41278 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/953cdf5eb2864dd0b7f130d8c0a41278 2023-03-21 11:56:58,341 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/953cdf5eb2864dd0b7f130d8c0a41278, entries=150, sequenceid=486, filesize=11.9 K 2023-03-21 11:56:58,342 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/C/e95a7ba93afe4ab8b0b99e1160522c89 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/e95a7ba93afe4ab8b0b99e1160522c89 2023-03-21 11:56:58,344 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/e95a7ba93afe4ab8b0b99e1160522c89, entries=150, sequenceid=486, filesize=11.9 K 2023-03-21 11:56:58,345 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=147.60 KB/151140 for 886ee010948b04e81bf5ede3c083ff8b in 448ms, sequenceid=486, compaction requested=true 2023-03-21 11:56:58,345 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 886ee010948b04e81bf5ede3c083ff8b: 2023-03-21 11:56:58,345 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:58,345 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:56:58,345 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:56:58,345 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:58,345 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:56:58,345 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:56:58,346 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:58,346 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:58,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:58,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:58,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:58,346 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:58,346 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:56:58,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:58,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:58,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:58,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:58,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:58,347 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:56:58,347 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4fb71c52[Count = 0] remaining members to acquire global barrier 2023-03-21 11:56:58,347 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:56:58,347 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:58,348 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:58,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:58,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:58,348 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:56:58,348 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:58,348 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:56:58,348 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:56:58,348 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:56:58,349 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:56:58,349 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:58,349 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:58,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:58,349 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:56:58,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:58,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:58,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:58,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:58,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:58,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:58,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:58,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:58,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:58,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:56:58,350 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:56:58,350 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:56:58,351 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:56:58,351 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:56:58,351 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:56:58,351 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:56:58,351 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:58,351 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:58,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:58,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:58,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:58,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:58,351 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:58,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:58,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:58,352 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:58,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:58,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:58,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:58,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:58,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:58,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:58,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:58,353 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:58,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:58,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:58,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:58,354 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:58,354 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:58,354 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:58,354 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:58,354 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:58,354 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:58,354 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:56:58,354 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:58,354 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:58,354 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:56:58,354 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:56:58,354 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:58,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:58,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:58,354 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:58,354 DEBUG [Thread-1446] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:56:58,354 DEBUG [Thread-1446] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:56:58,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:58,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:58,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(9158): Flush requested on 886ee010948b04e81bf5ede3c083ff8b 2023-03-21 11:56:58,395 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 886ee010948b04e81bf5ede3c083ff8b 3/3 column families, dataSize=154.31 KB heapSize=437.39 KB 2023-03-21 11:56:58,395 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 886ee010948b04e81bf5ede3c083ff8b, store=A 2023-03-21 11:56:58,395 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-21 11:56:58,395 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 886ee010948b04e81bf5ede3c083ff8b, store=B 2023-03-21 11:56:58,396 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-21 11:56:58,396 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 886ee010948b04e81bf5ede3c083ff8b, store=C 2023-03-21 11:56:58,396 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-21 11:56:58,402 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=593 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/A/a77e8f47cca04c8f8c4908bae8dd3355 2023-03-21 11:56:58,406 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:58,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41730 deadline: 1679399878402, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:58,406 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:58,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41768 deadline: 1679399878402, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:58,406 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:58,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 264 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41780 deadline: 1679399878402, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:58,454 DEBUG [Thread-1446] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:56:58,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:56:58,456 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:56:58,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:56:58,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:56:58,457 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:56:58,457 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:58,457 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:56:58,457 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:56:58,458 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:58,458 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:58,458 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:58,458 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:58,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:58,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:58,459 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:58,459 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:56:58,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:58,459 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:58,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:56:58,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:58,459 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:56:58,459 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:56:58,459 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:56:58,460 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:56:58,460 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:58,460 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:56:58,460 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:56:58,460 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:56:58,460 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:58,460 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. started... 2023-03-21 11:56:58,460 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. as already flushing 2023-03-21 11:56:58,460 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:58,460 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:56:58,460 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:56:58,460 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:58,460 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:56:58,460 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:56:58,461 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:58,461 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:58,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:58,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:58,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:58,461 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:58,461 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:56:58,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:58,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:58,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:58,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:58,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:58,462 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:56:58,462 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:56:58,462 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6cff4e9[Count = 0] remaining members to acquire global barrier 2023-03-21 11:56:58,462 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:58,463 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:58,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:58,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:58,463 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:56:58,463 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:56:58,463 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:58,463 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:56:58,463 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:56:58,464 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:58,464 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:56:58,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:58,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:58,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:58,464 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:58,464 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:56:58,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:58,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:58,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:58,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:58,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:58,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:58,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:58,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:56:58,465 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:56:58,465 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:56:58,466 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:56:58,466 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:56:58,466 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:56:58,466 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:56:58,466 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:58,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:58,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:58,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:58,466 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:58,467 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:58,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:58,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:58,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:58,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:58,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:58,467 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:58,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:58,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:58,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:58,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:58,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:58,468 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:58,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:58,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:58,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:58,470 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:58,470 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:58,470 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:58,470 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:58,470 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:58,470 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:56:58,470 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:56:58,470 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:58,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:58,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:58,470 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:58,470 DEBUG [Thread-1446] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:56:58,470 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:58,471 DEBUG [Thread-1446] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:56:58,471 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:58,471 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:58,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:58,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:58,505 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:58,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41826 deadline: 1679399878502, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:58,506 DEBUG [Thread-1440] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8142 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b., hostname=jenkins-hbase20.apache.org,34369,1679399732030, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-21 11:56:58,509 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:58,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41768 deadline: 1679399878507, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:58,510 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:58,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41730 deadline: 1679399878507, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:58,510 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:58,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 266 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41780 deadline: 1679399878507, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:58,571 DEBUG [Thread-1446] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:56:58,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:56:58,572 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:56:58,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:56:58,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:56:58,574 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:56:58,574 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:58,574 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:56:58,574 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:56:58,574 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:58,574 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:58,575 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:58,575 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:58,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:58,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:58,578 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:58,578 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:56:58,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:58,578 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:58,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:56:58,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:58,579 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:56:58,579 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:56:58,579 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:56:58,579 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:56:58,579 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:58,579 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:56:58,579 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:56:58,580 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:56:58,580 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:58,580 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. started... 2023-03-21 11:56:58,580 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. as already flushing 2023-03-21 11:56:58,580 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:58,580 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:56:58,580 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:56:58,580 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:58,580 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:56:58,580 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:56:58,581 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:58,581 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:58,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:58,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:58,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:58,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:58,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:58,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:58,582 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:58,582 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:56:58,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:58,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:58,582 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:56:58,582 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@634b7cd2[Count = 0] remaining members to acquire global barrier 2023-03-21 11:56:58,582 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:56:58,582 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:58,583 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:58,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:58,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:58,583 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:56:58,583 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:56:58,583 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:56:58,583 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:58,583 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:56:58,584 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:58,584 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:56:58,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:58,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:58,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:58,584 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:58,584 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:56:58,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:58,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:58,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:58,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:58,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:58,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:58,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:58,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:56:58,586 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:56:58,586 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:56:58,586 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:56:58,586 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:56:58,586 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:56:58,586 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:56:58,587 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:58,587 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:58,587 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:58,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:58,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:58,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:58,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:58,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:58,587 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:58,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:58,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:58,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:58,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:58,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:58,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:58,588 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:58,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:58,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:58,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:58,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:58,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:58,590 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:58,590 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:58,590 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:58,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:58,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:58,590 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:56:58,590 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:56:58,590 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:58,590 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:58,590 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:58,590 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:58,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:58,590 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:56:58,590 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:58,590 DEBUG [Thread-1446] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:56:58,590 DEBUG [Thread-1446] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:56:58,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:58,590 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:58,590 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:58,628 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:58,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41744 deadline: 1679399878626, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:58,690 DEBUG [Thread-1446] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:56:58,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:56:58,692 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:56:58,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:56:58,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:56:58,694 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:56:58,694 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:58,694 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:56:58,694 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:56:58,694 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:58,694 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:58,695 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:58,695 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:58,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:58,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:58,695 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:58,695 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:56:58,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:58,695 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:58,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:56:58,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:58,696 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:56:58,696 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:56:58,696 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:56:58,696 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:58,696 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:56:58,696 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:56:58,696 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:56:58,696 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:58,696 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. started... 2023-03-21 11:56:58,696 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. as already flushing 2023-03-21 11:56:58,696 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:58,696 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:56:58,696 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:56:58,696 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:58,696 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:56:58,696 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:56:58,697 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:58,697 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:58,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:58,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:58,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:58,697 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:58,697 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:56:58,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:58,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:58,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:58,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:58,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:58,698 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:56:58,699 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:56:58,699 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@55fb5f0[Count = 0] remaining members to acquire global barrier 2023-03-21 11:56:58,699 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:58,699 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:58,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:58,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:58,699 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:56:58,699 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:56:58,699 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:56:58,699 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:58,699 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:56:58,700 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:58,700 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:56:58,700 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:58,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:58,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:58,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:58,700 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:56:58,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:58,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:58,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:58,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:58,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:58,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:58,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:58,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:56:58,702 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:56:58,702 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:56:58,702 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:56:58,702 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:56:58,702 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:56:58,702 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:56:58,703 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:58,703 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:58,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:58,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:58,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:58,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:58,703 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:58,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:58,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:58,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:58,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:58,704 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:58,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:58,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:58,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:58,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:58,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:58,705 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:58,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:58,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:58,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:58,706 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:58,706 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:58,706 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:58,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:58,706 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:58,706 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:58,706 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:56:58,706 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:56:58,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:58,706 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:58,706 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:58,707 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:58,707 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:58,707 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:58,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:58,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:58,707 DEBUG [Thread-1446] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:56:58,707 DEBUG [Thread-1446] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:56:58,714 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:58,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41768 deadline: 1679399878711, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:58,714 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:58,714 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:58,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41730 deadline: 1679399878711, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:58,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 268 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41780 deadline: 1679399878712, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:58,807 DEBUG [Thread-1446] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:56:58,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:56:58,808 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:56:58,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:56:58,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:56:58,810 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:56:58,810 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:58,810 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:56:58,810 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:56:58,812 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:58,812 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=593 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/B/dae63057e46344bebf614d7df3f57ea4 2023-03-21 11:56:58,812 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:58,813 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:58,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:58,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:58,813 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:58,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:58,813 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:58,813 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:56:58,813 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:58,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:56:58,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:58,814 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:56:58,814 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:56:58,814 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:56:58,814 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:56:58,814 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:58,814 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:56:58,815 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:56:58,815 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:56:58,815 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:58,815 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. started... 2023-03-21 11:56:58,815 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. as already flushing 2023-03-21 11:56:58,815 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:58,815 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:56:58,815 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:56:58,815 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:58,815 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:56:58,815 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:56:58,816 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:58,816 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:58,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:58,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:58,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:58,817 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:58,817 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:56:58,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:58,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:58,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:58,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:58,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:58,818 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:56:58,818 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@72fd6c85[Count = 0] remaining members to acquire global barrier 2023-03-21 11:56:58,818 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:56:58,818 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:58,819 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:58,819 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:58,819 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:56:58,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:58,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:58,820 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:56:58,820 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:56:58,820 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:56:58,821 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:56:58,821 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:58,821 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:58,821 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:56:58,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:58,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:58,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:58,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:58,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:58,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:58,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:58,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:58,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:58,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:58,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:56:58,823 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:56:58,823 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:56:58,823 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:56:58,823 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:56:58,823 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:56:58,823 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:56:58,824 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=593 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/C/ef77dea4a1784839b14702320b3f3690 2023-03-21 11:56:58,825 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:58,825 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:58,825 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:58,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:58,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:58,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:58,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:58,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:58,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:58,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:58,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:58,827 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:58,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:58,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:58,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:58,828 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:58,828 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/A/a77e8f47cca04c8f8c4908bae8dd3355 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/a77e8f47cca04c8f8c4908bae8dd3355 2023-03-21 11:56:58,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:58,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:58,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:58,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:58,830 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:58,830 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:58,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:58,830 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:58,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:58,830 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:58,831 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:58,831 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:58,831 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:58,831 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:58,831 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:56:58,831 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:56:58,831 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:58,831 DEBUG [Thread-1446] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:56:58,831 DEBUG [Thread-1446] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:56:58,831 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:56:58,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:58,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:58,831 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:58,833 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/a77e8f47cca04c8f8c4908bae8dd3355, entries=200, sequenceid=593, filesize=14.3 K 2023-03-21 11:56:58,834 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/B/dae63057e46344bebf614d7df3f57ea4 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/dae63057e46344bebf614d7df3f57ea4 2023-03-21 11:56:58,837 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/dae63057e46344bebf614d7df3f57ea4, entries=150, sequenceid=593, filesize=11.9 K 2023-03-21 11:56:58,837 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/C/ef77dea4a1784839b14702320b3f3690 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/ef77dea4a1784839b14702320b3f3690 2023-03-21 11:56:58,842 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/ef77dea4a1784839b14702320b3f3690, entries=150, sequenceid=593, filesize=11.9 K 2023-03-21 11:56:58,843 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~154.31 KB/158010, heapSize ~437.34 KB/447840, currentSize=26.84 KB/27480 for 886ee010948b04e81bf5ede3c083ff8b in 448ms, sequenceid=593, compaction requested=true 2023-03-21 11:56:58,843 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 886ee010948b04e81bf5ede3c083ff8b: 2023-03-21 11:56:58,843 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-21 11:56:58,843 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-03-21 11:56:58,843 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-03-21 11:56:58,843 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-21 11:56:58,843 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-21 11:56:58,845 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 86014 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-03-21 11:56:58,845 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1912): 886ee010948b04e81bf5ede3c083ff8b/A is initiating minor compaction (all files) 2023-03-21 11:56:58,845 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 886ee010948b04e81bf5ede3c083ff8b/A in TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:58,845 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/5f32f0c676994684acb666ac96ab9d71, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/1b5080d7ef444b328fa937fa566802a5, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/a17b925816d2443f888acac99c38dcf8, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/1ae9ddb1370b4d26bf263ec31a0eb180, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/80d691407a124458bd61a335298416db, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/244c8cc84a734307aebcf39a362bedf3, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/a77e8f47cca04c8f8c4908bae8dd3355] into tmpdir=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp, totalSize=84.0 K 2023-03-21 11:56:58,846 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 83574 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-03-21 11:56:58,846 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 5f32f0c676994684acb666ac96ab9d71, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=298, earliestPutTs=1679399813800 2023-03-21 11:56:58,846 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1912): 886ee010948b04e81bf5ede3c083ff8b/B is initiating minor compaction (all files) 2023-03-21 11:56:58,846 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 886ee010948b04e81bf5ede3c083ff8b/B in TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:58,846 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/1a78dcc7492b4daf89506206e57a5d20, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/2933409c7e814eb687ac0f3336f0d780, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/c4e09e3793bd455bab52bd48d7cdeafc, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/aefe42cada67458d9522aacfe7037a42, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/09944657c93748f8a3cbbce23a1ddb8d, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/953cdf5eb2864dd0b7f130d8c0a41278, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/dae63057e46344bebf614d7df3f57ea4] into tmpdir=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp, totalSize=81.6 K 2023-03-21 11:56:58,846 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 1a78dcc7492b4daf89506206e57a5d20, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=298, earliestPutTs=1679399813800 2023-03-21 11:56:58,847 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 1b5080d7ef444b328fa937fa566802a5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=310, earliestPutTs=1679399814089 2023-03-21 11:56:58,847 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 2933409c7e814eb687ac0f3336f0d780, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=310, earliestPutTs=1679399814089 2023-03-21 11:56:58,847 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting a17b925816d2443f888acac99c38dcf8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=364, earliestPutTs=1679399814464 2023-03-21 11:56:58,847 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting c4e09e3793bd455bab52bd48d7cdeafc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=364, earliestPutTs=1679399814464 2023-03-21 11:56:58,848 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 1ae9ddb1370b4d26bf263ec31a0eb180, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=372, earliestPutTs=1679399815084 2023-03-21 11:56:58,848 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting aefe42cada67458d9522aacfe7037a42, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=372, earliestPutTs=1679399815084 2023-03-21 11:56:58,848 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 80d691407a124458bd61a335298416db, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=478, earliestPutTs=1679399815772 2023-03-21 11:56:58,848 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 09944657c93748f8a3cbbce23a1ddb8d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=478, earliestPutTs=1679399815772 2023-03-21 11:56:58,848 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 244c8cc84a734307aebcf39a362bedf3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=486, earliestPutTs=1679399816837 2023-03-21 11:56:58,848 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 953cdf5eb2864dd0b7f130d8c0a41278, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=486, earliestPutTs=1679399816837 2023-03-21 11:56:58,848 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting a77e8f47cca04c8f8c4908bae8dd3355, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=593, earliestPutTs=1679399818071 2023-03-21 11:56:58,849 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting dae63057e46344bebf614d7df3f57ea4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=593, earliestPutTs=1679399818071 2023-03-21 11:56:58,860 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] throttle.PressureAwareThroughputController(145): 886ee010948b04e81bf5ede3c083ff8b#A#compaction#255 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-21 11:56:58,861 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] throttle.PressureAwareThroughputController(145): 886ee010948b04e81bf5ede3c083ff8b#B#compaction#256 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-21 11:56:58,884 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/B/2d7b8088b347457aa870f1e40cc1b28e as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/2d7b8088b347457aa870f1e40cc1b28e 2023-03-21 11:56:58,898 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 886ee010948b04e81bf5ede3c083ff8b/B of 886ee010948b04e81bf5ede3c083ff8b into 2d7b8088b347457aa870f1e40cc1b28e(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-21 11:56:58,898 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 886ee010948b04e81bf5ede3c083ff8b: 2023-03-21 11:56:58,898 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b., storeName=886ee010948b04e81bf5ede3c083ff8b/B, priority=9, startTime=1679399818843; duration=0sec 2023-03-21 11:56:58,898 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-21 11:56:58,898 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-03-21 11:56:58,900 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 83574 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-03-21 11:56:58,900 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1912): 886ee010948b04e81bf5ede3c083ff8b/C is initiating minor compaction (all files) 2023-03-21 11:56:58,900 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 886ee010948b04e81bf5ede3c083ff8b/C in TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:58,900 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/b798803d08ba461c910d60a95eb91e95, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/d96a44630578478fbdc148c147373a97, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/14c19255426741d98459554ed87ffc70, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/5865371a0afd4e39ace91c13292aaa5a, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/3bbcf6077c8647cb91b582719621334f, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/e95a7ba93afe4ab8b0b99e1160522c89, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/ef77dea4a1784839b14702320b3f3690] into tmpdir=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp, totalSize=81.6 K 2023-03-21 11:56:58,901 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting b798803d08ba461c910d60a95eb91e95, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=298, earliestPutTs=1679399813800 2023-03-21 11:56:58,901 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting d96a44630578478fbdc148c147373a97, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=310, earliestPutTs=1679399814089 2023-03-21 11:56:58,901 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 14c19255426741d98459554ed87ffc70, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=364, earliestPutTs=1679399814464 2023-03-21 11:56:58,901 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 5865371a0afd4e39ace91c13292aaa5a, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=372, earliestPutTs=1679399815084 2023-03-21 11:56:58,902 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 3bbcf6077c8647cb91b582719621334f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=478, earliestPutTs=1679399815772 2023-03-21 11:56:58,902 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting e95a7ba93afe4ab8b0b99e1160522c89, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=486, earliestPutTs=1679399816837 2023-03-21 11:56:58,902 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting ef77dea4a1784839b14702320b3f3690, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=593, earliestPutTs=1679399818071 2023-03-21 11:56:58,915 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] throttle.PressureAwareThroughputController(145): 886ee010948b04e81bf5ede3c083ff8b#C#compaction#257 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-21 11:56:58,931 DEBUG [Thread-1446] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:56:58,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:56:58,934 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:56:58,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:56:58,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:56:58,938 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:56:58,938 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:58,938 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:56:58,938 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:56:58,938 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:58,938 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:58,940 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:58,940 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:58,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:58,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:58,940 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:58,940 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:56:58,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:58,940 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:58,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:56:58,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:58,941 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:56:58,941 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:56:58,941 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:56:58,941 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:56:58,941 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:58,941 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:56:58,941 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:56:58,941 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:56:58,941 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:58,941 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. started... 2023-03-21 11:56:58,941 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 886ee010948b04e81bf5ede3c083ff8b 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-03-21 11:56:58,942 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 886ee010948b04e81bf5ede3c083ff8b, store=A 2023-03-21 11:56:58,942 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:56:58,942 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 886ee010948b04e81bf5ede3c083ff8b, store=B 2023-03-21 11:56:58,942 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:56:58,942 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 886ee010948b04e81bf5ede3c083ff8b, store=C 2023-03-21 11:56:58,942 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:56:58,948 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=601 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/A/9e725d02022b48919e780ae768d95594 2023-03-21 11:56:58,958 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=601 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/B/699695417a1a4395a245bdf913962548 2023-03-21 11:56:59,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(9158): Flush requested on 886ee010948b04e81bf5ede3c083ff8b 2023-03-21 11:56:59,018 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. as already flushing 2023-03-21 11:56:59,035 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679399819029, max timestamp=1679399819033Num uniques 859; 2023-03-21 11:56:59,035 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-21 11:56:59,035 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679399819029, max timestamp=1679399819033Num uniques 859; 2023-03-21 11:56:59,035 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679399819029, max timestamp=1679399819033Num uniques 859; 2023-03-21 11:56:59,035 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-21 11:56:59,035 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-21 11:56:59,058 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:59,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41768 deadline: 1679399879056, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:59,059 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1679399819044, max timestamp=1679399819056Num uniques 1068; 2023-03-21 11:56:59,059 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1679399819044, max timestamp=1679399819056Num uniques 1068; 2023-03-21 11:56:59,059 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1679399819044, max timestamp=1679399819056Num uniques 1068; 2023-03-21 11:56:59,059 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-03-21 11:56:59,059 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-03-21 11:56:59,059 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-03-21 11:56:59,098 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1679399819072, max timestamp=1679399819084Num uniques 1027; 2023-03-21 11:56:59,098 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-03-21 11:56:59,099 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1679399819072, max timestamp=1679399819084Num uniques 1027; 2023-03-21 11:56:59,099 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-03-21 11:56:59,099 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1679399819072, max timestamp=1679399819084Num uniques 1027; 2023-03-21 11:56:59,099 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-03-21 11:56:59,134 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1679399819107, max timestamp=1679399819132Num uniques 1086; 2023-03-21 11:56:59,134 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=10806, suffix heap size=140912,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=236 2023-03-21 11:56:59,134 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1679399819107, max timestamp=1679399819132Num uniques 1086; 2023-03-21 11:56:59,134 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=10806, suffix heap size=140912,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=236 2023-03-21 11:56:59,134 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1679399819107, max timestamp=1679399819132Num uniques 1086; 2023-03-21 11:56:59,134 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=10806, suffix heap size=140912,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=236 2023-03-21 11:56:59,155 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13510, totalHeapSize=30920, min timestamp=1679399819145, max timestamp=1679399819153Num uniques 1095; 2023-03-21 11:56:59,155 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1679399819146, max timestamp=1679399819153Num uniques 1095; 2023-03-21 11:56:59,155 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=13510, suffix heap size=141848,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=295 2023-03-21 11:56:59,155 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=8930, suffix heap size=141848,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=195 2023-03-21 11:56:59,155 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1679399819146, max timestamp=1679399819153Num uniques 1095; 2023-03-21 11:56:59,156 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=8930, suffix heap size=141848,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=195 2023-03-21 11:56:59,177 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1679399819166, max timestamp=1679399819175Num uniques 1054; 2023-03-21 11:56:59,177 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=9340, suffix heap size=137584,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=204 2023-03-21 11:56:59,178 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1679399819166, max timestamp=1679399819175Num uniques 1154; 2023-03-21 11:56:59,178 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=52850, new segment data size=9340, suffix heap size=147984,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1154, new segment cells count=204 2023-03-21 11:56:59,178 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1679399819166, max timestamp=1679399819175Num uniques 1054; 2023-03-21 11:56:59,178 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=9340, suffix heap size=137584,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=204 2023-03-21 11:56:59,197 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=263, cellSize=12038, totalHeapSize=27592, min timestamp=1679399819185, max timestamp=1679399819195Num uniques 1063; 2023-03-21 11:56:59,197 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=12038, suffix heap size=138520,new segment heap size=27592 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=263 2023-03-21 11:56:59,197 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=263, cellSize=12038, totalHeapSize=27592, min timestamp=1679399819185, max timestamp=1679399819195Num uniques 1063; 2023-03-21 11:56:59,198 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=263, cellSize=12038, totalHeapSize=27592, min timestamp=1679399819185, max timestamp=1679399819195Num uniques 1063; 2023-03-21 11:56:59,198 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=12038, suffix heap size=138520,new segment heap size=27592 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=263 2023-03-21 11:56:59,198 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=12038, suffix heap size=138520,new segment heap size=27592 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=263 2023-03-21 11:56:59,238 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=272, cellSize=12452, totalHeapSize=28528, min timestamp=1679399819223, max timestamp=1679399819237Num uniques 1122; 2023-03-21 11:56:59,239 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=51382, new segment data size=12452, suffix heap size=144656,new segment heap size=28528 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1122, new segment cells count=272 2023-03-21 11:56:59,239 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=272, cellSize=12452, totalHeapSize=28528, min timestamp=1679399819223, max timestamp=1679399819237Num uniques 1122; 2023-03-21 11:56:59,239 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=51382, new segment data size=12452, suffix heap size=144656,new segment heap size=28528 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1122, new segment cells count=272 2023-03-21 11:56:59,239 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=272, cellSize=12452, totalHeapSize=28528, min timestamp=1679399819223, max timestamp=1679399819237Num uniques 1122; 2023-03-21 11:56:59,239 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=51382, new segment data size=12452, suffix heap size=144656,new segment heap size=28528 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1122, new segment cells count=272 2023-03-21 11:56:59,255 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:59,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41744 deadline: 1679399879251, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:59,256 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1679399819248, max timestamp=1679399819254Num uniques 1131; 2023-03-21 11:56:59,256 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=51796, new segment data size=10576, suffix heap size=145592,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1131, new segment cells count=231 2023-03-21 11:56:59,256 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1679399819248, max timestamp=1679399819254Num uniques 1131; 2023-03-21 11:56:59,256 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=51796, new segment data size=10576, suffix heap size=145592,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1131, new segment cells count=231 2023-03-21 11:56:59,257 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1679399819248, max timestamp=1679399819254Num uniques 1131; 2023-03-21 11:56:59,257 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=51796, new segment data size=10576, suffix heap size=145592,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1131, new segment cells count=231 2023-03-21 11:56:59,276 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=290, cellSize=13280, totalHeapSize=30400, min timestamp=1679399819260, max timestamp=1679399819274Num uniques 1090; 2023-03-21 11:56:59,276 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=13280, suffix heap size=141328,new segment heap size=30400 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=290 2023-03-21 11:56:59,276 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=290, cellSize=13280, totalHeapSize=30400, min timestamp=1679399819260, max timestamp=1679399819274Num uniques 1090; 2023-03-21 11:56:59,276 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=13280, suffix heap size=141328,new segment heap size=30400 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=290 2023-03-21 11:56:59,276 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=290, cellSize=13280, totalHeapSize=30400, min timestamp=1679399819260, max timestamp=1679399819274Num uniques 1090; 2023-03-21 11:56:59,276 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=13280, suffix heap size=141328,new segment heap size=30400 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=290 2023-03-21 11:56:59,302 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/A/079cfcdf286f46338c2b24429f370e57 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/079cfcdf286f46338c2b24429f370e57 2023-03-21 11:56:59,302 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:59,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 332 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41780 deadline: 1679399879298, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:59,303 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:59,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 261 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41768 deadline: 1679399879299, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:59,304 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1679399819297, max timestamp=1679399819301Num uniques 1149; 2023-03-21 11:56:59,304 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=52624, new segment data size=9114, suffix heap size=147464,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1149, new segment cells count=199 2023-03-21 11:56:59,304 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1679399819297, max timestamp=1679399819301Num uniques 1149; 2023-03-21 11:56:59,304 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=52624, new segment data size=9114, suffix heap size=147464,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1149, new segment cells count=199 2023-03-21 11:56:59,306 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1679399819297, max timestamp=1679399819301Num uniques 1149; 2023-03-21 11:56:59,306 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=52624, new segment data size=9114, suffix heap size=147464,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1149, new segment cells count=199 2023-03-21 11:56:59,309 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 886ee010948b04e81bf5ede3c083ff8b/A of 886ee010948b04e81bf5ede3c083ff8b into 079cfcdf286f46338c2b24429f370e57(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-21 11:56:59,309 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 886ee010948b04e81bf5ede3c083ff8b: 2023-03-21 11:56:59,309 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b., storeName=886ee010948b04e81bf5ede3c083ff8b/A, priority=9, startTime=1679399818843; duration=0sec 2023-03-21 11:56:59,309 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-21 11:56:59,332 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/C/ab0b80841a3d437dafed450b71ad4b71 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/ab0b80841a3d437dafed450b71ad4b71 2023-03-21 11:56:59,336 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 886ee010948b04e81bf5ede3c083ff8b/C of 886ee010948b04e81bf5ede3c083ff8b into ab0b80841a3d437dafed450b71ad4b71(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-21 11:56:59,336 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 886ee010948b04e81bf5ede3c083ff8b: 2023-03-21 11:56:59,336 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b., storeName=886ee010948b04e81bf5ede3c083ff8b/C, priority=9, startTime=1679399818843; duration=0sec 2023-03-21 11:56:59,336 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-21 11:56:59,359 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1679399819348, max timestamp=1679399819357Num uniques 1058; 2023-03-21 11:56:59,359 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=9520, suffix heap size=138000,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=208 2023-03-21 11:56:59,359 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1679399819348, max timestamp=1679399819357Num uniques 1058; 2023-03-21 11:56:59,359 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1679399819348, max timestamp=1679399819357Num uniques 1058; 2023-03-21 11:56:59,359 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=9520, suffix heap size=138000,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=208 2023-03-21 11:56:59,359 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=9520, suffix heap size=138000,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=208 2023-03-21 11:56:59,368 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=601 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/C/3d9d5993dcf4448aaa9fc36609b77c7d 2023-03-21 11:56:59,371 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/A/9e725d02022b48919e780ae768d95594 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/9e725d02022b48919e780ae768d95594 2023-03-21 11:56:59,374 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/9e725d02022b48919e780ae768d95594, entries=100, sequenceid=601, filesize=9.5 K 2023-03-21 11:56:59,374 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/B/699695417a1a4395a245bdf913962548 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/699695417a1a4395a245bdf913962548 2023-03-21 11:56:59,378 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/699695417a1a4395a245bdf913962548, entries=100, sequenceid=601, filesize=9.5 K 2023-03-21 11:56:59,378 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/C/3d9d5993dcf4448aaa9fc36609b77c7d as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/3d9d5993dcf4448aaa9fc36609b77c7d 2023-03-21 11:56:59,381 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/3d9d5993dcf4448aaa9fc36609b77c7d, entries=100, sequenceid=601, filesize=9.5 K 2023-03-21 11:56:59,382 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=134.18 KB/137400 for 886ee010948b04e81bf5ede3c083ff8b in 441ms, sequenceid=601, compaction requested=false 2023-03-21 11:56:59,382 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 886ee010948b04e81bf5ede3c083ff8b: 2023-03-21 11:56:59,382 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:59,382 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:56:59,382 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:56:59,382 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:59,382 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:56:59,382 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:56:59,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(9158): Flush requested on 886ee010948b04e81bf5ede3c083ff8b 2023-03-21 11:56:59,382 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 886ee010948b04e81bf5ede3c083ff8b 3/3 column families, dataSize=140.89 KB heapSize=400.31 KB 2023-03-21 11:56:59,383 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:59,383 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:59,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:59,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:59,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:59,384 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:59,384 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:56:59,384 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 886ee010948b04e81bf5ede3c083ff8b, store=A 2023-03-21 11:56:59,384 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-21 11:56:59,384 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 886ee010948b04e81bf5ede3c083ff8b, store=B 2023-03-21 11:56:59,384 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-21 11:56:59,384 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 886ee010948b04e81bf5ede3c083ff8b, store=C 2023-03-21 11:56:59,384 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-21 11:56:59,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:59,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:59,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:59,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:59,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:59,388 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:56:59,388 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b6ca68f[Count = 0] remaining members to acquire global barrier 2023-03-21 11:56:59,388 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:56:59,388 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:59,389 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:59,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:59,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:59,389 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:56:59,389 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:56:59,389 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:56:59,389 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:59,389 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:56:59,390 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:56:59,390 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:59,390 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:59,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:59,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:59,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:59,390 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:56:59,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:59,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:59,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:59,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:59,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:59,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:59,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:59,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:56:59,392 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:56:59,392 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:56:59,392 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:56:59,392 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:56:59,392 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:56:59,392 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:56:59,393 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:59,393 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:59,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:59,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:59,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:59,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:59,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:59,393 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:59,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:59,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:59,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:59,394 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:59,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:59,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:59,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:59,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:59,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:59,395 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:59,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:59,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:59,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:59,397 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:59,397 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:59,397 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:59,397 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:59,397 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:59,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:59,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:59,397 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:59,397 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:56:59,397 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:56:59,397 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:59,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:59,397 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:59,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:59,398 DEBUG [Thread-1446] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:56:59,398 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:59,398 DEBUG [Thread-1446] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:56:59,398 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:59,408 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:59,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 334 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41780 deadline: 1679399879403, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:59,408 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:59,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 263 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41768 deadline: 1679399879403, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:59,410 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:59,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41744 deadline: 1679399879408, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:59,410 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:59,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 313 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41730 deadline: 1679399879408, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:59,498 DEBUG [Thread-1446] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:56:59,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:56:59,499 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:56:59,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:56:59,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:56:59,500 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:56:59,500 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:59,500 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:56:59,500 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:56:59,501 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:59,501 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:59,502 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:59,502 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:59,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:59,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:59,502 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:59,502 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:56:59,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:59,502 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:59,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:56:59,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:59,502 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:56:59,503 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:56:59,503 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:56:59,503 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:56:59,503 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:59,503 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:56:59,505 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:56:59,505 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:56:59,505 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:59,505 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. started... 2023-03-21 11:56:59,505 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. as already flushing 2023-03-21 11:56:59,505 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:59,505 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:56:59,505 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:56:59,505 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:59,505 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:56:59,505 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:56:59,507 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:59,507 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:59,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:59,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:59,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:59,508 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:59,508 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:56:59,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:59,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:59,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:59,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:59,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:59,509 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:56:59,509 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3379e6dc[Count = 0] remaining members to acquire global barrier 2023-03-21 11:56:59,509 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:56:59,509 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:59,510 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:59,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:59,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:59,510 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:56:59,510 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:56:59,510 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:56:59,510 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:59,510 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:56:59,511 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:59,511 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:56:59,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:59,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:59,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:59,511 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:59,511 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:56:59,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:59,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:59,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:59,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:59,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:59,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:59,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:59,512 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:59,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 315 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41730 deadline: 1679399879511, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:59,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:56:59,513 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:59,513 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:56:59,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41744 deadline: 1679399879511, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:59,513 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:56:59,513 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:56:59,513 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:56:59,513 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:56:59,513 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:56:59,514 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:59,514 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:59,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:59,514 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:59,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:59,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:59,514 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:59,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:59,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:59,515 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:59,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:59,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:59,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:59,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:59,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:59,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:59,516 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:59,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:59,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:59,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:59,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:59,518 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:59,518 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:59,518 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:59,518 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:59,518 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:59,518 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:59,518 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:59,518 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:59,518 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:59,518 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:56:59,518 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:56:59,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:59,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:59,518 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:59,518 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:56:59,518 DEBUG [Thread-1446] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:56:59,519 DEBUG [Thread-1446] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:56:59,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:59,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:59,611 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:59,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 336 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41780 deadline: 1679399879610, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:59,612 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:59,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 265 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41768 deadline: 1679399879610, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:59,619 DEBUG [Thread-1446] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:56:59,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:56:59,620 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:56:59,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:56:59,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:56:59,621 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:56:59,621 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:59,621 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:56:59,621 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:56:59,622 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:59,622 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:59,622 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:59,622 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:59,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:59,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:59,622 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:59,623 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:56:59,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:59,623 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:59,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:56:59,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:59,623 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:56:59,623 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:56:59,623 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:56:59,623 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:56:59,623 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:59,624 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:56:59,624 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:56:59,624 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:56:59,624 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:59,624 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. started... 2023-03-21 11:56:59,624 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. as already flushing 2023-03-21 11:56:59,624 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:59,624 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:56:59,624 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:56:59,624 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:59,624 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:56:59,624 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:56:59,625 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:59,625 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:59,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:59,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:59,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:59,625 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:59,625 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:56:59,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:59,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:59,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:59,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:59,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:59,626 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:56:59,626 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b713576[Count = 0] remaining members to acquire global barrier 2023-03-21 11:56:59,626 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:56:59,626 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:59,627 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:59,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:59,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:59,627 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:56:59,627 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:56:59,627 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:56:59,627 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:59,627 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:56:59,628 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:59,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:59,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:59,628 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:56:59,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:59,628 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:59,628 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:56:59,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:59,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:59,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:59,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:59,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:59,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:59,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:59,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:56:59,629 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:56:59,629 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:56:59,630 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:56:59,630 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:56:59,630 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:56:59,630 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:56:59,630 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:59,630 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:59,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:59,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:59,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:59,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:59,631 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:59,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:59,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:59,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:59,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:59,631 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:59,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:59,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:59,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:59,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:59,632 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:59,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:59,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:59,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:59,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:59,634 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:59,634 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:59,634 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:59,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:59,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:59,634 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:59,634 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:56:59,634 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:56:59,634 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:59,634 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:59,634 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:59,634 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:59,634 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:56:59,634 DEBUG [Thread-1446] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:56:59,634 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:59,634 DEBUG [Thread-1446] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:56:59,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:59,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:59,634 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:59,714 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:59,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 317 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41730 deadline: 1679399879713, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:59,716 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:59,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 246 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41744 deadline: 1679399879714, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:59,735 DEBUG [Thread-1446] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:56:59,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:56:59,736 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:56:59,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:56:59,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:56:59,737 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:56:59,737 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:59,737 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:56:59,737 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:56:59,738 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:59,738 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:59,738 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:59,738 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:59,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:59,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:59,738 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:59,738 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:56:59,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:59,739 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:59,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:56:59,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:59,739 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:56:59,739 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:56:59,739 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:56:59,739 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:56:59,739 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:59,739 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:56:59,740 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:56:59,740 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:56:59,740 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:59,740 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. started... 2023-03-21 11:56:59,740 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. as already flushing 2023-03-21 11:56:59,740 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:59,740 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:56:59,740 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:56:59,740 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:59,740 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:56:59,740 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:56:59,741 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:59,741 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:59,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:59,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:59,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:59,741 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:59,741 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:56:59,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:59,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:59,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:59,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:59,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:59,742 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:56:59,742 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:56:59,742 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@52f1e1d9[Count = 0] remaining members to acquire global barrier 2023-03-21 11:56:59,742 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:59,743 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:59,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:59,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:59,743 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:56:59,743 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:56:59,743 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:56:59,743 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:59,743 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:56:59,744 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:59,744 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:56:59,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:59,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:59,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:59,744 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:59,744 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:56:59,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:59,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:59,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:59,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:59,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:59,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:59,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:59,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:56:59,745 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:56:59,745 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:56:59,745 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:56:59,745 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:56:59,745 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:56:59,745 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:56:59,746 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:59,746 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:59,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:59,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:59,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:59,746 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:59,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:59,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:59,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:59,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:59,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:59,746 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:59,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:59,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:59,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:59,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:59,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:59,747 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:59,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:59,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:59,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:59,748 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:59,748 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:59,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:59,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:59,749 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:59,749 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:59,749 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:59,749 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:59,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:59,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:59,749 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:59,749 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:56:59,749 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:56:59,749 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:59,749 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:59,749 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:59,749 DEBUG [Thread-1446] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:56:59,749 DEBUG [Thread-1446] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:56:59,796 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=830 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/A/9e0cf8129de34d0aac51d8b1895b94e5 2023-03-21 11:56:59,849 DEBUG [Thread-1446] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:56:59,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:56:59,851 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:56:59,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:56:59,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:56:59,852 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:56:59,852 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:59,852 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:56:59,852 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:56:59,852 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:59,852 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:59,853 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:59,853 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:59,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:59,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:59,854 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:59,854 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:56:59,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:59,854 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:59,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:56:59,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:59,855 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:56:59,855 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:56:59,855 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:56:59,855 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:56:59,855 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:59,855 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:56:59,855 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:56:59,855 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:56:59,855 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:59,855 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. started... 2023-03-21 11:56:59,855 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. as already flushing 2023-03-21 11:56:59,856 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:59,856 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:56:59,856 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:56:59,856 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:59,856 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:56:59,856 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:56:59,857 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:59,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:59,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:59,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:59,857 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:59,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:59,857 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:59,857 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:56:59,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:59,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:59,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:59,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:59,858 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:56:59,858 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3cc76dd6[Count = 0] remaining members to acquire global barrier 2023-03-21 11:56:59,858 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:56:59,858 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:59,859 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:59,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:59,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:59,859 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:56:59,859 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:56:59,859 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:56:59,859 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:59,859 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:56:59,860 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:56:59,860 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:59,860 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:59,860 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:56:59,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:59,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:59,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:59,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:59,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:59,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:59,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:59,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:59,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:59,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:59,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:56:59,862 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:56:59,862 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:56:59,862 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:56:59,862 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:56:59,862 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:56:59,862 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:56:59,863 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:59,863 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:59,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:59,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:59,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:59,863 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:59,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:59,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:59,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:59,863 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:59,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:59,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:59,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:59,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:59,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:59,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:59,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:59,864 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:59,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:59,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:59,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:59,865 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:59,865 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:59,865 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:59,865 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:56:59,865 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:56:59,865 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:59,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:59,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:59,865 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:59,866 DEBUG [Thread-1446] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:56:59,865 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:59,866 DEBUG [Thread-1446] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:56:59,866 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:56:59,866 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:59,866 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:59,866 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:59,866 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:59,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:59,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:59,914 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:59,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 267 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41768 deadline: 1679399879914, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:59,915 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:56:59,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 338 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41780 deadline: 1679399879914, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:59,966 DEBUG [Thread-1446] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:56:59,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:56:59,967 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:56:59,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:56:59,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:56:59,968 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:56:59,968 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:59,968 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:56:59,968 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:56:59,968 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:59,968 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:59,969 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:59,969 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:59,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:59,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:59,969 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:59,969 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:56:59,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:59,969 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:59,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:56:59,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:59,970 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:56:59,970 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:56:59,970 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:56:59,970 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:56:59,970 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:56:59,970 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:56:59,970 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:56:59,970 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:56:59,970 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:59,970 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. started... 2023-03-21 11:56:59,970 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. as already flushing 2023-03-21 11:56:59,971 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:56:59,971 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:56:59,971 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:56:59,971 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:59,971 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:56:59,971 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:56:59,971 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:59,971 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:59,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:59,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:59,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:59,972 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:59,972 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:56:59,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:59,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:59,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:59,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:59,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:59,973 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:56:59,973 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:56:59,973 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@40f42394[Count = 0] remaining members to acquire global barrier 2023-03-21 11:56:59,973 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:59,973 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:59,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:59,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:59,973 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:56:59,973 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:56:59,973 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:59,973 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:56:59,973 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:56:59,974 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:56:59,974 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:59,974 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:59,974 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:56:59,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:59,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:59,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:59,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:59,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:59,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:59,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:59,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:59,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:59,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:59,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:56:59,976 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:56:59,976 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:56:59,976 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:56:59,976 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:56:59,976 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:56:59,976 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:56:59,977 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:59,977 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:59,977 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:59,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:59,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:59,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:59,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:56:59,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:56:59,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:59,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:59,978 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:59,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:56:59,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:56:59,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:59,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:56:59,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:59,978 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:59,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:59,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:56:59,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:56:59,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:59,980 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:59,980 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:59,980 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:56:59,980 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:56:59,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:56:59,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:56:59,980 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:56:59,980 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:56:59,980 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:56:59,980 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:56:59,980 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:56:59,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:56:59,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:56:59,980 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:56:59,980 DEBUG [Thread-1446] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:56:59,980 DEBUG [Thread-1446] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:56:59,980 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:59,980 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:56:59,980 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:00,017 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:00,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 319 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41730 deadline: 1679399880016, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:00,020 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:00,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 248 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41744 deadline: 1679399880018, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:00,080 DEBUG [Thread-1446] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:00,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:00,082 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:57:00,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:57:00,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:57:00,083 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:57:00,083 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:00,083 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:57:00,083 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:57:00,083 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:00,083 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:00,084 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:00,084 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:00,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:00,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:00,084 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:00,084 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:57:00,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:00,084 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:00,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:57:00,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:00,085 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:57:00,085 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:57:00,085 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:57:00,085 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:57:00,085 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:00,085 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:57:00,085 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:57:00,085 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:57:00,085 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:57:00,085 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. started... 2023-03-21 11:57:00,085 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. as already flushing 2023-03-21 11:57:00,085 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:57:00,085 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:57:00,085 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:57:00,085 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:00,086 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:57:00,086 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:57:00,086 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:00,086 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:00,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:00,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:00,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:00,086 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:00,086 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:57:00,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:00,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:00,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:00,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:00,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:00,087 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:57:00,087 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:57:00,087 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ec7479[Count = 0] remaining members to acquire global barrier 2023-03-21 11:57:00,087 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:00,088 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:00,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:00,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:00,088 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:00,088 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:57:00,088 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:57:00,088 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:57:00,088 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:57:00,089 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:57:00,089 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:00,089 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:00,089 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:57:00,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:00,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:00,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:00,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:00,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:00,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:00,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:00,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:00,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:00,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:00,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:57:00,091 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:57:00,091 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:57:00,091 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:57:00,091 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:57:00,091 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:57:00,091 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:57:00,092 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:00,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:00,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:00,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:00,092 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:00,092 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:00,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:00,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:00,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:00,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:00,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:00,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:00,092 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:00,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:00,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:00,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:00,093 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:00,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:00,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:00,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:00,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:00,094 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:00,094 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:00,094 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:00,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:00,094 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:00,094 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:00,095 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:57:00,095 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:57:00,095 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:57:00,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:00,094 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:00,095 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:00,095 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:00,095 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:00,095 DEBUG [Thread-1446] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:57:00,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:00,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:00,095 DEBUG [Thread-1446] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:57:00,095 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:00,195 DEBUG [Thread-1446] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:00,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:00,197 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:57:00,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:57:00,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:57:00,198 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:57:00,198 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:00,198 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:57:00,198 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:57:00,198 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:00,198 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:00,199 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:00,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:00,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:00,199 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:00,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:00,199 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:00,200 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:57:00,200 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:00,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:57:00,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:00,200 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:57:00,200 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:57:00,200 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:57:00,200 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:57:00,200 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:00,200 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:57:00,200 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:57:00,200 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:57:00,200 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:57:00,201 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. started... 2023-03-21 11:57:00,201 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. as already flushing 2023-03-21 11:57:00,201 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:57:00,201 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:57:00,201 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:57:00,201 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:00,201 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:57:00,201 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:57:00,202 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:00,202 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:00,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:00,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:00,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:00,202 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:00,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:00,202 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:57:00,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:00,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:00,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:00,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:00,203 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:57:00,203 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5fceb2d4[Count = 0] remaining members to acquire global barrier 2023-03-21 11:57:00,203 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:57:00,203 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:00,204 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:00,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:00,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:00,204 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:57:00,204 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:57:00,204 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:57:00,204 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=830 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/B/4c31707dbddf4ba0a50b8b41a1afedbb 2023-03-21 11:57:00,204 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:57:00,205 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:00,204 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:00,205 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:57:00,205 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:57:00,205 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:57:00,205 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:57:00,205 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:57:00,205 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:57:00,205 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:57:00,205 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:57:00,206 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:00,206 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:00,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:00,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:00,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:00,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:00,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:00,206 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:00,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:00,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:00,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:00,207 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:00,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:00,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:00,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:00,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:00,207 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:00,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:00,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:00,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:00,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:00,209 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:00,209 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:00,209 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:00,209 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:00,209 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:57:00,209 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:57:00,209 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:00,209 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:00,209 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:00,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:00,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:00,209 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:00,209 DEBUG [Thread-1446] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:57:00,209 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:00,209 DEBUG [Thread-1446] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:57:00,209 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:00,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:00,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:00,210 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:57:00,310 DEBUG [Thread-1446] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:00,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:00,311 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:57:00,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:57:00,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:57:00,312 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:57:00,312 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:00,313 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:57:00,313 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:57:00,313 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:00,313 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:00,314 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:00,314 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:00,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:00,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:00,314 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:00,314 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:57:00,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:00,314 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:00,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:57:00,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:00,315 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:57:00,315 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:57:00,315 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:57:00,315 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:57:00,315 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:00,315 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:57:00,315 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:57:00,315 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:57:00,315 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:57:00,315 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. started... 2023-03-21 11:57:00,315 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. as already flushing 2023-03-21 11:57:00,315 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:57:00,315 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:57:00,315 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:57:00,315 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:00,315 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:57:00,315 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:57:00,316 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:00,316 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:00,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:00,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:00,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:00,316 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:00,316 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:57:00,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:00,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:00,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:00,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:00,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:00,317 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:57:00,318 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5860cd26[Count = 0] remaining members to acquire global barrier 2023-03-21 11:57:00,318 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:57:00,318 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:00,318 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:00,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:00,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:00,318 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:57:00,318 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:57:00,318 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:57:00,318 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:00,319 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:57:00,319 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:57:00,319 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:00,319 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:00,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:00,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:00,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:00,319 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:57:00,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:00,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:00,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:00,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:00,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:00,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:00,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:00,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:57:00,321 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:57:00,321 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:57:00,321 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:57:00,321 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:57:00,321 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:57:00,321 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:57:00,322 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:00,322 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:00,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:00,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:00,322 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:00,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:00,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:00,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:00,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:00,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:00,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:00,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:00,323 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:00,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:00,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:00,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:00,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:00,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:00,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:00,324 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:00,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:00,326 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:00,326 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:00,326 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:00,326 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:00,326 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:00,326 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:00,326 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:57:00,326 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:57:00,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:00,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:00,326 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:00,326 DEBUG [Thread-1446] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:57:00,326 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:00,326 DEBUG [Thread-1446] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:57:00,326 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:00,326 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:00,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:00,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:00,418 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:00,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 340 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41780 deadline: 1679399880416, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:00,419 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:00,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 269 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41768 deadline: 1679399880417, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:00,426 DEBUG [Thread-1446] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:00,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:00,428 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:57:00,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:57:00,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:57:00,429 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:57:00,429 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:00,429 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:57:00,429 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:57:00,429 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:00,430 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:00,430 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:00,430 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:00,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:00,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:00,430 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:00,430 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:57:00,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:00,431 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:00,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:57:00,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:00,431 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:57:00,431 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:57:00,431 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:57:00,431 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:57:00,431 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:00,431 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:57:00,432 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:57:00,432 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:57:00,432 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:57:00,432 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. started... 2023-03-21 11:57:00,432 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. as already flushing 2023-03-21 11:57:00,432 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:57:00,432 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:57:00,432 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:57:00,432 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:00,432 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:57:00,432 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:57:00,433 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:00,433 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:00,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:00,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:00,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:00,433 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:00,433 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:57:00,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:00,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:00,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:00,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:00,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:00,434 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:57:00,434 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3969ad3d[Count = 0] remaining members to acquire global barrier 2023-03-21 11:57:00,434 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:57:00,434 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:00,435 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:00,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:00,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:00,435 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:57:00,435 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:57:00,435 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:57:00,435 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:00,435 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:57:00,435 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:57:00,435 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:00,435 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:00,436 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:57:00,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:00,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:00,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:00,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:00,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:00,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:00,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:00,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:00,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:00,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:00,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:57:00,437 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:57:00,437 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:57:00,438 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:57:00,438 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:57:00,438 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:57:00,438 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:57:00,439 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:00,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:00,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:00,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:00,439 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:00,439 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:00,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:00,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:00,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:00,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:00,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:00,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:00,439 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:00,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:00,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:00,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:00,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:00,440 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:00,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:00,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:00,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:00,442 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:00,442 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:00,442 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:00,442 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:00,442 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:57:00,442 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:57:00,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:00,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:00,442 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:00,442 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:00,442 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:00,442 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:00,442 DEBUG [Thread-1446] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:57:00,442 DEBUG [Thread-1446] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:57:00,442 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:00,442 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:00,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:00,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:00,522 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:00,522 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:00,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 321 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41730 deadline: 1679399880520, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:00,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 250 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41744 deadline: 1679399880520, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:00,542 DEBUG [Thread-1446] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:00,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:00,544 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:57:00,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:57:00,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:57:00,545 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:57:00,545 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:00,545 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:57:00,545 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:57:00,545 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:00,545 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:00,546 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:00,546 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:00,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:00,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:00,546 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:00,546 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:57:00,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:00,547 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:00,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:57:00,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:00,547 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:57:00,547 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:57:00,547 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:57:00,547 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:57:00,547 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:00,547 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:57:00,547 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:57:00,547 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:57:00,547 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:57:00,547 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. started... 2023-03-21 11:57:00,547 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. as already flushing 2023-03-21 11:57:00,547 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:57:00,548 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:57:00,548 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:57:00,548 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:00,548 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:57:00,548 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:57:00,548 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:00,548 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:00,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:00,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:00,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:00,549 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:00,549 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:57:00,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:00,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:00,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:00,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:00,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:00,550 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:57:00,550 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:57:00,550 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@58c749d5[Count = 0] remaining members to acquire global barrier 2023-03-21 11:57:00,550 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:00,550 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:00,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:00,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:00,550 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:57:00,551 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:57:00,551 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:57:00,551 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:00,551 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:57:00,551 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:00,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:00,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:00,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:00,551 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:57:00,551 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:00,551 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:57:00,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:00,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:00,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:00,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:00,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:00,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:00,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:00,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:57:00,553 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:57:00,553 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:57:00,553 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:57:00,553 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:57:00,553 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:57:00,553 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:57:00,553 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:00,553 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:00,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:00,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:00,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:00,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:00,554 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:00,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:00,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:00,554 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:00,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:00,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:00,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:00,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:00,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:00,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:00,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:00,555 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:00,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:00,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:00,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:00,556 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:00,556 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:00,556 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:00,556 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:00,556 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:00,557 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:57:00,557 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:57:00,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:00,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:00,557 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:00,557 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:00,557 DEBUG [Thread-1446] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:57:00,557 DEBUG [Thread-1446] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:57:00,557 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:00,557 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:00,557 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:00,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:00,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:00,614 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=830 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/C/ec76054a69ab4638910f08b314e01ccf 2023-03-21 11:57:00,617 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/A/9e0cf8129de34d0aac51d8b1895b94e5 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/9e0cf8129de34d0aac51d8b1895b94e5 2023-03-21 11:57:00,620 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/9e0cf8129de34d0aac51d8b1895b94e5, entries=250, sequenceid=830, filesize=16.7 K 2023-03-21 11:57:00,620 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/B/4c31707dbddf4ba0a50b8b41a1afedbb as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/4c31707dbddf4ba0a50b8b41a1afedbb 2023-03-21 11:57:00,623 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/4c31707dbddf4ba0a50b8b41a1afedbb, entries=150, sequenceid=830, filesize=11.9 K 2023-03-21 11:57:00,624 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/C/ec76054a69ab4638910f08b314e01ccf as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/ec76054a69ab4638910f08b314e01ccf 2023-03-21 11:57:00,627 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/ec76054a69ab4638910f08b314e01ccf, entries=150, sequenceid=830, filesize=11.9 K 2023-03-21 11:57:00,628 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~420.89 KB/430992, currentSize=33.54 KB/34350 for 886ee010948b04e81bf5ede3c083ff8b in 1246ms, sequenceid=830, compaction requested=true 2023-03-21 11:57:00,628 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 886ee010948b04e81bf5ede3c083ff8b: 2023-03-21 11:57:00,628 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-21 11:57:00,628 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-21 11:57:00,628 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-21 11:57:00,628 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-21 11:57:00,628 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-21 11:57:00,629 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39948 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-21 11:57:00,629 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1912): 886ee010948b04e81bf5ede3c083ff8b/A is initiating minor compaction (all files) 2023-03-21 11:57:00,629 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 886ee010948b04e81bf5ede3c083ff8b/A in TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:57:00,629 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/079cfcdf286f46338c2b24429f370e57, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/9e725d02022b48919e780ae768d95594, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/9e0cf8129de34d0aac51d8b1895b94e5] into tmpdir=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp, totalSize=39.0 K 2023-03-21 11:57:00,629 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35068 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-21 11:57:00,629 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1912): 886ee010948b04e81bf5ede3c083ff8b/B is initiating minor compaction (all files) 2023-03-21 11:57:00,629 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 886ee010948b04e81bf5ede3c083ff8b/B in TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:57:00,629 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/2d7b8088b347457aa870f1e40cc1b28e, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/699695417a1a4395a245bdf913962548, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/4c31707dbddf4ba0a50b8b41a1afedbb] into tmpdir=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp, totalSize=34.2 K 2023-03-21 11:57:00,629 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 079cfcdf286f46338c2b24429f370e57, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=593, earliestPutTs=1679399818071 2023-03-21 11:57:00,630 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 2d7b8088b347457aa870f1e40cc1b28e, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=593, earliestPutTs=1679399818071 2023-03-21 11:57:00,630 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 9e725d02022b48919e780ae768d95594, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=601, earliestPutTs=1679399818401 2023-03-21 11:57:00,630 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 699695417a1a4395a245bdf913962548, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=601, earliestPutTs=1679399818401 2023-03-21 11:57:00,630 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 4c31707dbddf4ba0a50b8b41a1afedbb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=830, earliestPutTs=1679399819377 2023-03-21 11:57:00,630 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 9e0cf8129de34d0aac51d8b1895b94e5, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=830, earliestPutTs=1679399819375 2023-03-21 11:57:00,635 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] throttle.PressureAwareThroughputController(145): 886ee010948b04e81bf5ede3c083ff8b#A#compaction#264 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-21 11:57:00,635 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] throttle.PressureAwareThroughputController(145): 886ee010948b04e81bf5ede3c083ff8b#B#compaction#265 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-21 11:57:00,646 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/B/7b30d880e9e74a0388ee5e3275f7b2fe as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/7b30d880e9e74a0388ee5e3275f7b2fe 2023-03-21 11:57:00,650 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 886ee010948b04e81bf5ede3c083ff8b/B of 886ee010948b04e81bf5ede3c083ff8b into 7b30d880e9e74a0388ee5e3275f7b2fe(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-21 11:57:00,650 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 886ee010948b04e81bf5ede3c083ff8b: 2023-03-21 11:57:00,650 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b., storeName=886ee010948b04e81bf5ede3c083ff8b/B, priority=13, startTime=1679399820628; duration=0sec 2023-03-21 11:57:00,650 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-21 11:57:00,650 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-21 11:57:00,651 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35068 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-21 11:57:00,651 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1912): 886ee010948b04e81bf5ede3c083ff8b/C is initiating minor compaction (all files) 2023-03-21 11:57:00,651 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 886ee010948b04e81bf5ede3c083ff8b/C in TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:57:00,651 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/ab0b80841a3d437dafed450b71ad4b71, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/3d9d5993dcf4448aaa9fc36609b77c7d, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/ec76054a69ab4638910f08b314e01ccf] into tmpdir=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp, totalSize=34.2 K 2023-03-21 11:57:00,651 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting ab0b80841a3d437dafed450b71ad4b71, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=593, earliestPutTs=1679399818071 2023-03-21 11:57:00,652 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 3d9d5993dcf4448aaa9fc36609b77c7d, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=601, earliestPutTs=1679399818401 2023-03-21 11:57:00,652 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting ec76054a69ab4638910f08b314e01ccf, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=830, earliestPutTs=1679399819377 2023-03-21 11:57:00,656 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] throttle.PressureAwareThroughputController(145): 886ee010948b04e81bf5ede3c083ff8b#C#compaction#266 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-21 11:57:00,657 DEBUG [Thread-1446] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:00,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:00,659 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:57:00,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:57:00,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:57:00,662 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:57:00,662 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:00,662 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:57:00,662 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:57:00,662 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:00,663 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:00,663 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:00,663 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:00,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:00,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:00,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:00,666 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:00,666 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:57:00,666 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:00,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:57:00,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:00,667 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:57:00,667 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:57:00,667 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:57:00,667 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:57:00,667 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:00,667 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:57:00,667 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:57:00,667 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:57:00,667 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:57:00,667 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. started... 2023-03-21 11:57:00,667 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 886ee010948b04e81bf5ede3c083ff8b 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-03-21 11:57:00,667 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 886ee010948b04e81bf5ede3c083ff8b, store=A 2023-03-21 11:57:00,668 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:57:00,668 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 886ee010948b04e81bf5ede3c083ff8b, store=B 2023-03-21 11:57:00,668 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:57:00,668 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 886ee010948b04e81bf5ede3c083ff8b, store=C 2023-03-21 11:57:00,668 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:57:01,042 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/A/f97b4e45be594059a9f5d12a31b350db as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/f97b4e45be594059a9f5d12a31b350db 2023-03-21 11:57:01,046 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 886ee010948b04e81bf5ede3c083ff8b/A of 886ee010948b04e81bf5ede3c083ff8b into f97b4e45be594059a9f5d12a31b350db(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-21 11:57:01,046 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 886ee010948b04e81bf5ede3c083ff8b: 2023-03-21 11:57:01,046 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b., storeName=886ee010948b04e81bf5ede3c083ff8b/A, priority=13, startTime=1679399820628; duration=0sec 2023-03-21 11:57:01,046 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-21 11:57:01,067 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/C/79184bdb6a6a441c916b8406c42aa45f as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/79184bdb6a6a441c916b8406c42aa45f 2023-03-21 11:57:01,070 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 886ee010948b04e81bf5ede3c083ff8b/C of 886ee010948b04e81bf5ede3c083ff8b into 79184bdb6a6a441c916b8406c42aa45f(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-21 11:57:01,070 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 886ee010948b04e81bf5ede3c083ff8b: 2023-03-21 11:57:01,070 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b., storeName=886ee010948b04e81bf5ede3c083ff8b/C, priority=13, startTime=1679399820628; duration=0sec 2023-03-21 11:57:01,070 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-21 11:57:01,075 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=839 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/A/1cf41e793aa945318a86580c2920f453 2023-03-21 11:57:01,086 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=839 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/B/5781b45f8dab41ef8bff9369a1f3bcbc 2023-03-21 11:57:01,107 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=839 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/C/a9b7a83dd4284862929a431fa24f8411 2023-03-21 11:57:01,111 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/A/1cf41e793aa945318a86580c2920f453 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/1cf41e793aa945318a86580c2920f453 2023-03-21 11:57:01,114 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/1cf41e793aa945318a86580c2920f453, entries=150, sequenceid=839, filesize=11.9 K 2023-03-21 11:57:01,114 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/B/5781b45f8dab41ef8bff9369a1f3bcbc as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/5781b45f8dab41ef8bff9369a1f3bcbc 2023-03-21 11:57:01,122 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/5781b45f8dab41ef8bff9369a1f3bcbc, entries=150, sequenceid=839, filesize=11.9 K 2023-03-21 11:57:01,123 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/C/a9b7a83dd4284862929a431fa24f8411 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/a9b7a83dd4284862929a431fa24f8411 2023-03-21 11:57:01,127 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/a9b7a83dd4284862929a431fa24f8411, entries=150, sequenceid=839, filesize=11.9 K 2023-03-21 11:57:01,128 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=0 B/0 for 886ee010948b04e81bf5ede3c083ff8b in 461ms, sequenceid=839, compaction requested=false 2023-03-21 11:57:01,128 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 886ee010948b04e81bf5ede3c083ff8b: 2023-03-21 11:57:01,128 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:57:01,128 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:57:01,129 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:57:01,129 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,129 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:57:01,129 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:57:01,130 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,130 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:01,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:01,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:01,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:01,131 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:01,131 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:57:01,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:01,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:01,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:01,133 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:57:01,133 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@72b5080b[Count = 0] remaining members to acquire global barrier 2023-03-21 11:57:01,133 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:57:01,133 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:01,133 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:01,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:01,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:01,134 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:57:01,134 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:57:01,134 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:57:01,134 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,134 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:57:01,134 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:57:01,134 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:01,135 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:57:01,135 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:01,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:01,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:01,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:01,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:01,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:01,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:01,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:57:01,138 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:57:01,138 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:57:01,138 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:57:01,138 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:57:01,138 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:57:01,138 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:57:01,139 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:01,139 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:01,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:01,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:01,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:01,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:01,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:01,139 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:01,140 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:01,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:01,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:01,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:01,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:01,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:01,141 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:01,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:01,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:01,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,143 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:01,143 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:01,143 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:01,143 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:57:01,143 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:57:01,143 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:01,143 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:01,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:01,144 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:01,144 DEBUG [Thread-1446] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:57:01,144 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,144 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:01,144 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:01,144 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:01,144 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:57:01,144 DEBUG [Thread-1446] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:57:01,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:01,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:01,244 DEBUG [Thread-1446] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:01,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:01,246 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:57:01,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:57:01,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:57:01,247 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:57:01,247 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:01,247 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:57:01,247 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:57:01,247 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:01,247 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:01,248 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,248 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:01,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:01,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:01,248 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,248 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:57:01,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:01,248 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:01,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:57:01,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:01,249 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:57:01,249 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:57:01,249 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:57:01,249 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:57:01,249 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:01,249 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:57:01,249 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:57:01,249 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:57:01,249 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:57:01,249 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. started... 2023-03-21 11:57:01,249 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 886ee010948b04e81bf5ede3c083ff8b: 2023-03-21 11:57:01,249 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:57:01,249 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:57:01,249 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:57:01,249 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,250 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:57:01,250 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:57:01,250 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,250 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:01,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:01,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:01,251 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:01,251 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:57:01,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:01,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:01,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:01,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:01,252 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:57:01,252 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ce2f286[Count = 0] remaining members to acquire global barrier 2023-03-21 11:57:01,252 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:57:01,252 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:01,252 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:01,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:01,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:01,252 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:57:01,252 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:57:01,253 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:57:01,253 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,253 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:57:01,253 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:57:01,253 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:01,253 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,253 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:57:01,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:01,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:01,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:01,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:01,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:01,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:01,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:01,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:57:01,255 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:57:01,255 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:57:01,255 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:57:01,255 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:57:01,255 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:57:01,255 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:57:01,256 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:01,256 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:01,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:01,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:01,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:01,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:01,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:01,256 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:01,257 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:01,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:01,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:01,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:01,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:01,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:01,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:01,258 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:01,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:01,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,260 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,260 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:01,260 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:01,260 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:01,260 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:57:01,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:01,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:01,260 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:01,260 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:57:01,260 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:57:01,260 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:01,261 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:01,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:01,261 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:01,261 DEBUG [Thread-1446] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:57:01,261 DEBUG [Thread-1446] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:57:01,261 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:01,261 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:01,361 DEBUG [Thread-1446] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:01,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:01,362 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:57:01,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:57:01,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:57:01,363 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:57:01,363 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:01,363 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:57:01,363 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:57:01,364 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:01,364 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:01,364 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:01,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:01,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:01,364 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,365 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,365 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:57:01,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:01,365 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:01,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:57:01,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:01,365 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:57:01,365 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:57:01,366 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:57:01,366 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:57:01,366 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:01,366 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:57:01,366 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:57:01,366 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:57:01,366 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:57:01,366 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. started... 2023-03-21 11:57:01,366 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 886ee010948b04e81bf5ede3c083ff8b: 2023-03-21 11:57:01,366 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:57:01,366 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:57:01,366 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:57:01,366 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,366 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:57:01,366 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:57:01,367 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,367 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:01,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:01,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:01,367 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:01,367 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:57:01,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:01,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:01,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:01,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:01,368 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:57:01,368 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5816b262[Count = 0] remaining members to acquire global barrier 2023-03-21 11:57:01,368 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:57:01,368 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:01,369 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:01,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:01,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:01,369 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:57:01,369 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:57:01,369 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:57:01,369 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,369 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:57:01,369 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,370 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:57:01,370 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:01,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:01,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:01,370 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:57:01,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:01,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:01,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:01,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:01,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:01,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:57:01,371 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:57:01,371 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:57:01,371 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:57:01,371 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:57:01,372 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:57:01,372 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:57:01,372 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:01,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:01,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:01,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:01,373 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:01,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:01,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:01,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:01,373 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,373 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:01,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:01,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:01,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:01,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:01,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:01,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:01,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,374 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:01,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:01,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,376 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:01,376 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,376 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:01,376 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:01,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:01,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:01,376 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,376 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:57:01,376 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:57:01,376 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:01,376 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:01,376 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:01,376 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:01,376 DEBUG [Thread-1446] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:57:01,376 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:01,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:01,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:01,376 DEBUG [Thread-1446] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:57:01,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(9158): Flush requested on 886ee010948b04e81bf5ede3c083ff8b 2023-03-21 11:57:01,444 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 886ee010948b04e81bf5ede3c083ff8b 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-21 11:57:01,445 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 886ee010948b04e81bf5ede3c083ff8b, store=A 2023-03-21 11:57:01,445 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:57:01,445 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 886ee010948b04e81bf5ede3c083ff8b, store=B 2023-03-21 11:57:01,445 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:57:01,445 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 886ee010948b04e81bf5ede3c083ff8b, store=C 2023-03-21 11:57:01,445 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:57:01,450 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=854 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/A/79b296a0b7894b4db082af2ec1460102 2023-03-21 11:57:01,469 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:01,469 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679399821458, max timestamp=1679399821468Num uniques 859; 2023-03-21 11:57:01,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 285 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41768 deadline: 1679399881468, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,470 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-21 11:57:01,470 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679399821458, max timestamp=1679399821468Num uniques 859; 2023-03-21 11:57:01,470 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-21 11:57:01,477 DEBUG [Thread-1446] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:01,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:01,478 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:57:01,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:57:01,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:57:01,479 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:57:01,479 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:01,479 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:57:01,479 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:57:01,480 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:01,480 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:01,480 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,480 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:01,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:01,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:01,480 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,481 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:57:01,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:01,481 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:01,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:57:01,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:01,481 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:57:01,481 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:57:01,481 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:57:01,481 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:57:01,481 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:01,481 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:57:01,481 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:57:01,481 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:57:01,482 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:57:01,482 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. started... 2023-03-21 11:57:01,482 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. as already flushing 2023-03-21 11:57:01,482 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:57:01,482 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:57:01,482 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:57:01,482 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,482 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:57:01,482 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:57:01,483 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:01,483 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:01,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:01,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:01,483 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:01,484 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:57:01,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:01,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:01,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:01,484 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:57:01,484 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3afdc154[Count = 0] remaining members to acquire global barrier 2023-03-21 11:57:01,484 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:57:01,484 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:01,485 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:01,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:01,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:01,485 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:57:01,485 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:57:01,485 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:57:01,485 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,485 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:57:01,486 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:01,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:01,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:01,486 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:57:01,486 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:01,486 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:57:01,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:01,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:01,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:01,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:01,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:57:01,488 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:57:01,488 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:57:01,488 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:57:01,488 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:57:01,488 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:57:01,488 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:57:01,488 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:01,488 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:01,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:01,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:01,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:01,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:01,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:01,488 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:01,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:01,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:01,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:01,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:01,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:01,489 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:01,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:01,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:01,490 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:01,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,492 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:01,492 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:57:01,492 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:57:01,492 DEBUG [Thread-1446] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:57:01,492 DEBUG [Thread-1446] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:57:01,493 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,493 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:01,493 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:01,493 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,493 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:01,493 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:01,494 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:01,494 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:01,494 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:01,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:01,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:01,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:01,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:01,497 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:01,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 364 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41780 deadline: 1679399881496, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,526 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:01,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 252 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41744 deadline: 1679399881525, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,528 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:01,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 323 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41730 deadline: 1679399881527, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,572 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:01,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 287 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41768 deadline: 1679399881570, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,592 DEBUG [Thread-1446] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:01,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:01,594 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:57:01,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:57:01,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:57:01,595 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:57:01,595 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:01,595 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:57:01,595 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:57:01,595 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:01,595 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:01,596 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:01,596 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:01,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:01,596 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,596 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:57:01,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:01,596 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:01,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:57:01,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:01,596 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:57:01,597 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:57:01,597 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:57:01,597 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:57:01,597 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:01,597 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:57:01,597 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:57:01,597 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:57:01,597 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:57:01,597 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. started... 2023-03-21 11:57:01,597 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. as already flushing 2023-03-21 11:57:01,597 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:57:01,597 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:57:01,597 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:57:01,597 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,597 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:57:01,597 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:57:01,598 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,598 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:01,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:01,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:01,598 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:01,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:01,598 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:57:01,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:01,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:01,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:01,599 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:57:01,599 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:57:01,599 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:01,599 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@60d27350[Count = 0] remaining members to acquire global barrier 2023-03-21 11:57:01,600 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:01,600 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:01,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:01,600 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,600 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:57:01,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:01,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 366 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41780 deadline: 1679399881598, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,600 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:57:01,600 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:57:01,600 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:57:01,601 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,601 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:57:01,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:01,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:01,601 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:01,601 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:57:01,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:01,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:01,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:01,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:01,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:01,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:57:01,603 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:57:01,603 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:57:01,603 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:57:01,603 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:57:01,603 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:57:01,603 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:57:01,604 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:01,604 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:01,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:01,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:01,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:01,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:01,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:01,604 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:01,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:01,604 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:01,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:01,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:01,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:01,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:01,605 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:01,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:01,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:01,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,607 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,607 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:01,607 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:01,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:01,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:01,607 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:57:01,608 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:57:01,607 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:01,607 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:01,608 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:57:01,608 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:01,608 DEBUG [Thread-1446] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:57:01,608 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:01,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:01,608 DEBUG [Thread-1446] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:57:01,608 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:01,608 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:01,608 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:01,708 DEBUG [Thread-1446] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:01,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:01,710 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:57:01,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:57:01,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:57:01,711 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:57:01,711 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:01,711 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:57:01,711 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:57:01,711 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:01,711 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:01,712 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:01,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:01,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:01,712 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:01,712 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,712 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:57:01,712 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:01,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:57:01,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:01,713 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:57:01,713 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:57:01,713 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:57:01,713 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:57:01,713 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:01,713 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:57:01,713 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:57:01,713 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:57:01,713 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:57:01,713 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. started... 2023-03-21 11:57:01,713 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. as already flushing 2023-03-21 11:57:01,713 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:57:01,713 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:57:01,713 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:57:01,713 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,713 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:57:01,713 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:57:01,714 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:01,714 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:01,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:01,714 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:01,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:01,714 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:57:01,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:01,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:01,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:01,715 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:57:01,716 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:57:01,716 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b9fcb0e[Count = 0] remaining members to acquire global barrier 2023-03-21 11:57:01,716 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:01,716 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:01,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:01,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:01,716 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:57:01,716 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:57:01,716 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:57:01,716 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,716 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:57:01,717 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:57:01,717 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,717 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:01,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:01,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:01,717 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:57:01,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:01,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:01,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:01,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:01,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:01,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:57:01,719 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:57:01,719 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:57:01,719 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:57:01,719 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:57:01,719 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:57:01,719 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:57:01,720 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:01,720 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:01,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:01,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:01,720 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:01,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:01,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:01,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:01,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:01,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:01,720 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:01,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:01,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:01,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:01,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:01,721 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:01,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:01,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,723 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,723 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:01,723 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:01,723 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:01,723 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:01,723 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:01,723 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:01,723 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:57:01,723 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:57:01,723 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:01,723 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:01,723 DEBUG [Thread-1446] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:57:01,723 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:01,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:01,723 DEBUG [Thread-1446] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:57:01,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:01,723 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:57:01,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:01,776 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:01,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 289 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41768 deadline: 1679399881774, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,803 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:01,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 368 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41780 deadline: 1679399881802, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,823 DEBUG [Thread-1446] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:01,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:01,825 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:57:01,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:57:01,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:57:01,826 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:57:01,826 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:01,826 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:57:01,826 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:57:01,826 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:01,826 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:01,827 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:01,827 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:01,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:01,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:01,828 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:01,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:57:01,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:01,828 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:57:01,828 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:57:01,828 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:57:01,828 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:57:01,828 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:01,828 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,828 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:57:01,828 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:57:01,829 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:57:01,829 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:57:01,829 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:57:01,829 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. started... 2023-03-21 11:57:01,829 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. as already flushing 2023-03-21 11:57:01,829 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:57:01,829 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:57:01,829 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:57:01,829 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,829 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:57:01,829 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:57:01,830 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,830 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:01,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:01,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:01,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:01,830 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:01,830 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:57:01,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:01,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:01,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:01,832 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:57:01,832 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:57:01,832 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3677e7bd[Count = 0] remaining members to acquire global barrier 2023-03-21 11:57:01,832 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:01,832 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:01,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:01,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:01,833 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:57:01,833 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:57:01,833 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:57:01,833 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,833 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:57:01,833 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:01,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:01,833 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:57:01,833 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:01,834 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:57:01,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:01,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:01,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:01,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:01,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:01,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:57:01,835 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:57:01,835 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:57:01,835 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:57:01,835 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:57:01,835 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:57:01,835 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:57:01,836 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:01,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:01,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:01,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:01,836 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:01,836 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:01,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:01,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:01,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:01,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:01,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:01,837 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:01,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:01,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:01,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:01,838 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:01,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:01,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,839 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:01,839 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:01,839 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:01,839 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:01,839 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:57:01,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:01,839 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:57:01,840 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:57:01,839 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:01,840 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:01,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:01,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:01,840 DEBUG [Thread-1446] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:57:01,840 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,840 DEBUG [Thread-1446] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:57:01,840 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:01,840 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:01,840 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:01,857 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=854 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/B/b67b11fbc0bf45f9b9d6a81bd6bf7cec 2023-03-21 11:57:01,940 DEBUG [Thread-1446] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:01,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:01,941 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:57:01,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:57:01,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:57:01,942 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:57:01,942 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:01,942 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:57:01,943 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:57:01,943 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:01,943 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:01,943 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,944 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:01,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:01,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:01,944 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,944 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:57:01,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:01,944 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:01,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:57:01,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:01,944 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:57:01,945 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:57:01,945 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:57:01,945 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:57:01,945 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:01,945 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:57:01,945 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:57:01,945 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:57:01,945 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:57:01,945 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. started... 2023-03-21 11:57:01,945 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. as already flushing 2023-03-21 11:57:01,945 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:57:01,945 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:57:01,945 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:57:01,945 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,945 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:57:01,945 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:57:01,946 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,946 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:01,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:01,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:01,946 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:01,946 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:57:01,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:01,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:01,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:01,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:01,947 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:57:01,947 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:57:01,947 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@62d47746[Count = 0] remaining members to acquire global barrier 2023-03-21 11:57:01,947 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:01,948 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:01,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:01,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:01,948 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:57:01,948 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:57:01,948 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,948 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:57:01,948 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:57:01,949 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:01,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:01,949 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:57:01,949 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:01,949 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:57:01,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:01,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:01,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:01,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:01,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:01,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:57:01,951 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:57:01,951 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:57:01,951 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:57:01,951 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:57:01,951 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:57:01,951 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:57:01,952 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:01,952 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:01,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:01,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:01,952 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:01,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:01,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:01,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:01,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:01,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:01,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:01,952 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:01,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:01,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:01,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:01,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,953 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:01,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:01,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,957 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:01,957 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:01,957 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:57:01,957 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:57:01,957 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,957 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:01,957 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:01,957 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:01,957 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:01,957 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:01,957 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:01,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:01,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:01,957 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:01,957 DEBUG [Thread-1446] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:57:01,957 DEBUG [Thread-1446] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:57:01,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:01,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:01,958 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:57:02,057 DEBUG [Thread-1446] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:02,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:02,059 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:57:02,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:57:02,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:57:02,060 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:57:02,060 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:02,060 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:57:02,060 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:57:02,060 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:02,060 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:02,062 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:02,062 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:02,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:02,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:02,062 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:02,062 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:57:02,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:02,062 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:02,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:57:02,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:02,063 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:57:02,063 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:57:02,063 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:57:02,063 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:57:02,063 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:02,063 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:57:02,063 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:57:02,063 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:57:02,063 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:57:02,063 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. started... 2023-03-21 11:57:02,063 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. as already flushing 2023-03-21 11:57:02,063 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:57:02,063 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:57:02,063 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:57:02,063 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:02,063 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:57:02,063 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:57:02,065 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:02,065 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:02,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:02,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:02,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:02,065 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:02,065 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:57:02,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:02,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:02,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:02,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:02,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:02,066 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:57:02,066 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:57:02,066 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@116fc0bf[Count = 0] remaining members to acquire global barrier 2023-03-21 11:57:02,066 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:02,067 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:02,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:02,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:02,067 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:57:02,067 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:57:02,067 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:57:02,067 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:02,067 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:57:02,068 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:02,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:02,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:02,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:02,068 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:57:02,068 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:02,068 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:57:02,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:02,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:02,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:02,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:02,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:02,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:02,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:02,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:57:02,070 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:57:02,070 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:57:02,070 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:57:02,070 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:57:02,070 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:57:02,070 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:57:02,071 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:02,071 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:02,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:02,071 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:02,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:02,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:02,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:02,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:02,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:02,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:02,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:02,076 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:02,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:02,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:02,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:02,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:02,076 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:02,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:02,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:02,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:02,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:02,078 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:02,078 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:02,078 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:02,078 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:02,078 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:02,078 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:57:02,078 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:57:02,078 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:57:02,078 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:02,078 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:02,078 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:02,078 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:02,078 DEBUG [Thread-1446] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:57:02,078 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:02,078 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:02,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 291 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41768 deadline: 1679399882077, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:02,078 DEBUG [Thread-1446] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:57:02,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:02,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:02,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:02,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:02,108 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:02,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 370 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41780 deadline: 1679399882105, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:02,178 DEBUG [Thread-1446] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:02,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:02,180 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:57:02,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:57:02,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:57:02,181 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:57:02,181 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:02,181 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:57:02,181 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:57:02,181 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:02,181 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:02,182 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:02,182 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:02,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:02,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:02,182 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:02,182 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:57:02,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:02,183 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:02,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:57:02,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:02,183 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:57:02,183 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:57:02,183 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:57:02,183 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:57:02,183 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:02,183 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:57:02,183 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:57:02,183 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:57:02,183 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:57:02,183 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. started... 2023-03-21 11:57:02,183 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. as already flushing 2023-03-21 11:57:02,183 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:57:02,183 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:57:02,183 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:57:02,184 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:02,184 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:57:02,184 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:57:02,184 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:02,184 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:02,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:02,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:02,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:02,185 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:02,185 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:57:02,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:02,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:02,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:02,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:02,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:02,185 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:57:02,185 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:57:02,185 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@340dae27[Count = 0] remaining members to acquire global barrier 2023-03-21 11:57:02,185 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:02,186 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:02,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:02,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:02,186 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:57:02,186 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:02,186 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:57:02,186 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:57:02,186 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:57:02,187 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:02,187 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:57:02,187 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:02,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:02,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:02,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:02,187 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:57:02,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:02,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:02,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:02,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:02,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:02,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:02,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:02,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:57:02,188 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:57:02,188 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:57:02,188 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:57:02,188 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:57:02,188 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:57:02,188 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:57:02,189 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:02,189 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:02,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:02,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:02,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:02,189 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:02,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:02,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:02,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:02,189 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:02,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:02,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:02,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:02,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:02,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:02,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:02,190 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:02,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:02,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:02,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:02,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:02,192 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:02,192 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:02,192 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:02,192 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:57:02,192 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:57:02,192 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:57:02,192 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:02,192 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:02,192 DEBUG [Thread-1446] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:57:02,192 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:02,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:02,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:02,192 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:02,192 DEBUG [Thread-1446] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:57:02,192 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:02,192 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:02,192 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:02,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:02,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:02,265 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=854 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/C/84df4756430f42cfa1bdec9ea224c1bc 2023-03-21 11:57:02,269 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/A/79b296a0b7894b4db082af2ec1460102 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/79b296a0b7894b4db082af2ec1460102 2023-03-21 11:57:02,272 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/79b296a0b7894b4db082af2ec1460102, entries=200, sequenceid=854, filesize=14.3 K 2023-03-21 11:57:02,273 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/B/b67b11fbc0bf45f9b9d6a81bd6bf7cec as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/b67b11fbc0bf45f9b9d6a81bd6bf7cec 2023-03-21 11:57:02,275 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/b67b11fbc0bf45f9b9d6a81bd6bf7cec, entries=150, sequenceid=854, filesize=11.9 K 2023-03-21 11:57:02,276 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/C/84df4756430f42cfa1bdec9ea224c1bc as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/84df4756430f42cfa1bdec9ea224c1bc 2023-03-21 11:57:02,279 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/84df4756430f42cfa1bdec9ea224c1bc, entries=150, sequenceid=854, filesize=11.9 K 2023-03-21 11:57:02,279 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=125.23 KB/128240 for 886ee010948b04e81bf5ede3c083ff8b in 835ms, sequenceid=854, compaction requested=true 2023-03-21 11:57:02,279 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 886ee010948b04e81bf5ede3c083ff8b: 2023-03-21 11:57:02,280 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-21 11:57:02,280 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-21 11:57:02,280 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-21 11:57:02,280 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-21 11:57:02,280 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-21 11:57:02,280 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 40054 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-21 11:57:02,280 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1912): 886ee010948b04e81bf5ede3c083ff8b/A is initiating minor compaction (all files) 2023-03-21 11:57:02,280 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37614 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-21 11:57:02,280 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 886ee010948b04e81bf5ede3c083ff8b/A in TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:57:02,280 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1912): 886ee010948b04e81bf5ede3c083ff8b/B is initiating minor compaction (all files) 2023-03-21 11:57:02,280 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 886ee010948b04e81bf5ede3c083ff8b/B in TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:57:02,280 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/f97b4e45be594059a9f5d12a31b350db, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/1cf41e793aa945318a86580c2920f453, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/79b296a0b7894b4db082af2ec1460102] into tmpdir=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp, totalSize=39.1 K 2023-03-21 11:57:02,281 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/7b30d880e9e74a0388ee5e3275f7b2fe, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/5781b45f8dab41ef8bff9369a1f3bcbc, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/b67b11fbc0bf45f9b9d6a81bd6bf7cec] into tmpdir=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp, totalSize=36.7 K 2023-03-21 11:57:02,281 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting f97b4e45be594059a9f5d12a31b350db, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=830, earliestPutTs=1679399819377 2023-03-21 11:57:02,281 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 7b30d880e9e74a0388ee5e3275f7b2fe, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=830, earliestPutTs=1679399819377 2023-03-21 11:57:02,281 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 1cf41e793aa945318a86580c2920f453, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=839, earliestPutTs=1679399819391 2023-03-21 11:57:02,281 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 5781b45f8dab41ef8bff9369a1f3bcbc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=839, earliestPutTs=1679399819391 2023-03-21 11:57:02,281 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 79b296a0b7894b4db082af2ec1460102, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=854, earliestPutTs=1679399821427 2023-03-21 11:57:02,281 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting b67b11fbc0bf45f9b9d6a81bd6bf7cec, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=854, earliestPutTs=1679399821428 2023-03-21 11:57:02,288 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] throttle.PressureAwareThroughputController(145): 886ee010948b04e81bf5ede3c083ff8b#B#compaction#273 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-21 11:57:02,289 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] throttle.PressureAwareThroughputController(145): 886ee010948b04e81bf5ede3c083ff8b#A#compaction#274 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-21 11:57:02,292 DEBUG [Thread-1446] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:02,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:02,294 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:57:02,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:57:02,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:57:02,295 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:57:02,295 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:02,295 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:57:02,295 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:57:02,295 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:02,295 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:02,296 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:02,296 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:02,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:02,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:02,296 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:02,296 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:57:02,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:02,298 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:02,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:57:02,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:02,300 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:57:02,301 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:57:02,301 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:57:02,301 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:57:02,301 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/A/72b0243279e047f9a9bf5744cf54c9ae as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/72b0243279e047f9a9bf5744cf54c9ae 2023-03-21 11:57:02,301 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:02,301 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:57:02,301 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:57:02,301 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:57:02,301 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:57:02,301 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. started... 2023-03-21 11:57:02,301 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 886ee010948b04e81bf5ede3c083ff8b 3/3 column families, dataSize=125.23 KB heapSize=363.39 KB 2023-03-21 11:57:02,301 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 886ee010948b04e81bf5ede3c083ff8b, store=A 2023-03-21 11:57:02,301 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-21 11:57:02,301 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 886ee010948b04e81bf5ede3c083ff8b, store=B 2023-03-21 11:57:02,301 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-21 11:57:02,301 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 886ee010948b04e81bf5ede3c083ff8b, store=C 2023-03-21 11:57:02,301 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-21 11:57:02,305 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 886ee010948b04e81bf5ede3c083ff8b/A of 886ee010948b04e81bf5ede3c083ff8b into 72b0243279e047f9a9bf5744cf54c9ae(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-21 11:57:02,305 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 886ee010948b04e81bf5ede3c083ff8b: 2023-03-21 11:57:02,305 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b., storeName=886ee010948b04e81bf5ede3c083ff8b/A, priority=13, startTime=1679399822279; duration=0sec 2023-03-21 11:57:02,305 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-21 11:57:02,305 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-21 11:57:02,307 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37614 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-21 11:57:02,307 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1912): 886ee010948b04e81bf5ede3c083ff8b/C is initiating minor compaction (all files) 2023-03-21 11:57:02,307 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 886ee010948b04e81bf5ede3c083ff8b/C in TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:57:02,307 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/79184bdb6a6a441c916b8406c42aa45f, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/a9b7a83dd4284862929a431fa24f8411, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/84df4756430f42cfa1bdec9ea224c1bc] into tmpdir=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp, totalSize=36.7 K 2023-03-21 11:57:02,307 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 79184bdb6a6a441c916b8406c42aa45f, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=830, earliestPutTs=1679399819377 2023-03-21 11:57:02,308 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting a9b7a83dd4284862929a431fa24f8411, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=839, earliestPutTs=1679399819391 2023-03-21 11:57:02,308 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 84df4756430f42cfa1bdec9ea224c1bc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=854, earliestPutTs=1679399821428 2023-03-21 11:57:02,313 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=885 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/A/1614fedec4ab41cb8dced9a8f44a8285 2023-03-21 11:57:02,316 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] throttle.PressureAwareThroughputController(145): 886ee010948b04e81bf5ede3c083ff8b#C#compaction#276 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-21 11:57:02,322 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/C/a8cc952ac7e849168507119a6dcc6e34 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/a8cc952ac7e849168507119a6dcc6e34 2023-03-21 11:57:02,323 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=885 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/B/355d56e4f1e7467292a6de61768a2cf6 2023-03-21 11:57:02,327 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 886ee010948b04e81bf5ede3c083ff8b/C of 886ee010948b04e81bf5ede3c083ff8b into a8cc952ac7e849168507119a6dcc6e34(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-21 11:57:02,327 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 886ee010948b04e81bf5ede3c083ff8b: 2023-03-21 11:57:02,327 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b., storeName=886ee010948b04e81bf5ede3c083ff8b/C, priority=13, startTime=1679399822280; duration=0sec 2023-03-21 11:57:02,327 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-21 11:57:02,340 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=62.62 KB at sequenceid=885 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/C/14fd3fc9d69244e1813a7dd87dc799ab 2023-03-21 11:57:02,343 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/A/1614fedec4ab41cb8dced9a8f44a8285 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/1614fedec4ab41cb8dced9a8f44a8285 2023-03-21 11:57:02,349 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/1614fedec4ab41cb8dced9a8f44a8285, entries=150, sequenceid=885, filesize=11.9 K 2023-03-21 11:57:02,349 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/B/355d56e4f1e7467292a6de61768a2cf6 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/355d56e4f1e7467292a6de61768a2cf6 2023-03-21 11:57:02,354 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/355d56e4f1e7467292a6de61768a2cf6, entries=150, sequenceid=885, filesize=11.9 K 2023-03-21 11:57:02,354 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/C/14fd3fc9d69244e1813a7dd87dc799ab as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/14fd3fc9d69244e1813a7dd87dc799ab 2023-03-21 11:57:02,357 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/14fd3fc9d69244e1813a7dd87dc799ab, entries=150, sequenceid=885, filesize=11.9 K 2023-03-21 11:57:02,358 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~125.23 KB/128240, heapSize ~363.34 KB/372064, currentSize=0 B/0 for 886ee010948b04e81bf5ede3c083ff8b in 57ms, sequenceid=885, compaction requested=false 2023-03-21 11:57:02,358 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 886ee010948b04e81bf5ede3c083ff8b: 2023-03-21 11:57:02,358 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:57:02,358 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:57:02,358 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:57:02,358 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:02,358 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:57:02,358 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:57:02,360 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:02,360 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:02,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:02,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:02,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:02,360 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:02,360 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:57:02,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:02,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:02,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:02,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:02,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:02,362 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:57:02,362 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3dceed14[Count = 0] remaining members to acquire global barrier 2023-03-21 11:57:02,362 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:57:02,363 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:02,363 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:02,363 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:02,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:02,363 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:57:02,363 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:57:02,363 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:57:02,363 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:02,363 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:57:02,364 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:57:02,364 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:02,364 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:57:02,364 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:02,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:02,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:02,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:02,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:02,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:02,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:02,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:02,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:02,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:02,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:02,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:57:02,367 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:57:02,367 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:57:02,367 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:57:02,367 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:57:02,367 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:57:02,367 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:57:02,369 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:02,369 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:02,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:02,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:02,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:02,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:02,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:02,369 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:02,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:02,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:02,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:02,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:02,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:02,370 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:02,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:02,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:02,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:02,372 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:02,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:02,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:02,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:02,374 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:02,374 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:02,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:02,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:02,374 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:02,374 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:57:02,374 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:57:02,374 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:02,374 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:02,374 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:02,374 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:02,374 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:02,374 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:02,374 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:02,374 DEBUG [Thread-1446] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:57:02,375 DEBUG [Thread-1446] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:57:02,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:02,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:02,399 DEBUG [Thread-1451] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3fba9cd6 to 127.0.0.1:61761 2023-03-21 11:57:02,399 DEBUG [Thread-1451] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-21 11:57:02,399 DEBUG [Thread-1447] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7e9921a7 to 127.0.0.1:61761 2023-03-21 11:57:02,399 DEBUG [Thread-1447] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-21 11:57:02,399 DEBUG [Thread-1455] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x073dacc0 to 127.0.0.1:61761 2023-03-21 11:57:02,399 DEBUG [Thread-1455] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-21 11:57:02,399 DEBUG [Thread-1449] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5ffcf146 to 127.0.0.1:61761 2023-03-21 11:57:02,400 DEBUG [Thread-1449] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-21 11:57:02,400 DEBUG [Thread-1453] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5e897a83 to 127.0.0.1:61761 2023-03-21 11:57:02,400 DEBUG [Thread-1453] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-21 11:57:02,475 DEBUG [Thread-1446] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:02,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:02,584 DEBUG [Thread-1442] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2dbba21f to 127.0.0.1:61761 2023-03-21 11:57:02,584 DEBUG [Thread-1442] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-21 11:57:02,612 DEBUG [Thread-1444] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x54ba13aa to 127.0.0.1:61761 2023-03-21 11:57:02,612 DEBUG [Thread-1444] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-21 11:57:02,701 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/B/c74e27680f30448abc03fb71786ec292 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/c74e27680f30448abc03fb71786ec292 2023-03-21 11:57:02,710 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 886ee010948b04e81bf5ede3c083ff8b/B of 886ee010948b04e81bf5ede3c083ff8b into c74e27680f30448abc03fb71786ec292(size=13.0 K), total size for store is 24.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-21 11:57:02,710 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 886ee010948b04e81bf5ede3c083ff8b: 2023-03-21 11:57:02,710 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b., storeName=886ee010948b04e81bf5ede3c083ff8b/B, priority=13, startTime=1679399822280; duration=0sec 2023-03-21 11:57:02,710 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-21 11:57:03,533 DEBUG [Thread-1438] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5620345b to 127.0.0.1:61761 2023-03-21 11:57:03,533 DEBUG [Thread-1438] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-21 11:57:03,534 DEBUG [Thread-1436] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6298fe55 to 127.0.0.1:61761 2023-03-21 11:57:03,534 DEBUG [Thread-1436] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-21 11:57:08,549 DEBUG [Thread-1440] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x76e08050 to 127.0.0.1:61761 2023-03-21 11:57:08,549 DEBUG [Thread-1440] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-21 11:57:08,550 INFO [Listener at localhost.localdomain/37475] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-03-21 11:57:08,550 INFO [Listener at localhost.localdomain/37475] hbase.AcidGuaranteesTestTool(394): wrote 194 2023-03-21 11:57:08,550 INFO [Listener at localhost.localdomain/37475] hbase.AcidGuaranteesTestTool(394): wrote 123 2023-03-21 11:57:08,551 INFO [Listener at localhost.localdomain/37475] hbase.AcidGuaranteesTestTool(394): wrote 58 2023-03-21 11:57:08,551 INFO [Listener at localhost.localdomain/37475] hbase.AcidGuaranteesTestTool(394): wrote 160 2023-03-21 11:57:08,551 INFO [Listener at localhost.localdomain/37475] hbase.AcidGuaranteesTestTool(394): wrote 247 2023-03-21 11:57:08,551 INFO [Listener at localhost.localdomain/37475] hbase.AcidGuaranteesTestTool(396): Readers: 2023-03-21 11:57:08,552 INFO [Listener at localhost.localdomain/37475] hbase.AcidGuaranteesTestTool(398): read 5938 2023-03-21 11:57:08,552 INFO [Listener at localhost.localdomain/37475] hbase.AcidGuaranteesTestTool(398): read 5827 2023-03-21 11:57:08,552 INFO [Listener at localhost.localdomain/37475] hbase.AcidGuaranteesTestTool(398): read 5875 2023-03-21 11:57:08,552 INFO [Listener at localhost.localdomain/37475] hbase.AcidGuaranteesTestTool(398): read 5920 2023-03-21 11:57:08,552 INFO [Listener at localhost.localdomain/37475] hbase.AcidGuaranteesTestTool(398): read 5827 2023-03-21 11:57:08,553 INFO [Listener at localhost.localdomain/37475] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-03-21 11:57:08,553 INFO [Listener at localhost.localdomain/37475] client.ConnectionImplementation(1959): Closing master protocol: MasterService 2023-03-21 11:57:08,553 DEBUG [Listener at localhost.localdomain/37475] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x05fbdc11 to 127.0.0.1:61761 2023-03-21 11:57:08,553 DEBUG [Listener at localhost.localdomain/37475] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-21 11:57:08,554 INFO [Listener at localhost.localdomain/37475] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-03-21 11:57:08,554 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.HMaster$11(2418): Client=jenkins//148.251.75.209 disable TestAcidGuarantees 2023-03-21 11:57:08,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure2.ProcedureExecutor(1029): Stored pid=31, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-03-21 11:57:08,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-03-21 11:57:08,558 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679399828558"}]},"ts":"1679399828558"} 2023-03-21 11:57:08,558 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-03-21 11:57:08,559 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-03-21 11:57:08,560 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=886ee010948b04e81bf5ede3c083ff8b, UNASSIGN}] 2023-03-21 11:57:08,561 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=886ee010948b04e81bf5ede3c083ff8b, UNASSIGN 2023-03-21 11:57:08,561 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=886ee010948b04e81bf5ede3c083ff8b, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:08,562 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679399828561"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1679399828561"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679399828561"}]},"ts":"1679399828561"} 2023-03-21 11:57:08,562 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=33, ppid=32, state=RUNNABLE; CloseRegionProcedure 886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030}] 2023-03-21 11:57:08,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-03-21 11:57:08,715 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 886ee010948b04e81bf5ede3c083ff8b 2023-03-21 11:57:08,715 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 886ee010948b04e81bf5ede3c083ff8b, disabling compactions & flushes 2023-03-21 11:57:08,715 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:57:08,715 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:57:08,715 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. after waiting 0 ms 2023-03-21 11:57:08,715 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:57:08,715 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing 886ee010948b04e81bf5ede3c083ff8b 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-03-21 11:57:08,716 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 886ee010948b04e81bf5ede3c083ff8b, store=A 2023-03-21 11:57:08,716 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:57:08,716 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 886ee010948b04e81bf5ede3c083ff8b, store=B 2023-03-21 11:57:08,716 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:57:08,716 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 886ee010948b04e81bf5ede3c083ff8b, store=C 2023-03-21 11:57:08,716 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:57:08,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-03-21 11:57:09,130 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=896 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/A/e162bfc5c689471786861cb06ecb51fd 2023-03-21 11:57:09,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-03-21 11:57:09,546 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=896 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/B/fe5a0f28fe534719ae86613f54fb24cf 2023-03-21 11:57:09,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-03-21 11:57:09,958 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=896 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/C/dcd528dbdb03425f996d2cd53feb199d 2023-03-21 11:57:09,967 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/A/e162bfc5c689471786861cb06ecb51fd as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/e162bfc5c689471786861cb06ecb51fd 2023-03-21 11:57:09,972 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/e162bfc5c689471786861cb06ecb51fd, entries=150, sequenceid=896, filesize=11.9 K 2023-03-21 11:57:09,973 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/B/fe5a0f28fe534719ae86613f54fb24cf as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/fe5a0f28fe534719ae86613f54fb24cf 2023-03-21 11:57:09,976 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/fe5a0f28fe534719ae86613f54fb24cf, entries=150, sequenceid=896, filesize=11.9 K 2023-03-21 11:57:09,977 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/.tmp/C/dcd528dbdb03425f996d2cd53feb199d as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/dcd528dbdb03425f996d2cd53feb199d 2023-03-21 11:57:09,980 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/dcd528dbdb03425f996d2cd53feb199d, entries=150, sequenceid=896, filesize=11.9 K 2023-03-21 11:57:09,981 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=0 B/0 for 886ee010948b04e81bf5ede3c083ff8b in 1266ms, sequenceid=896, compaction requested=true 2023-03-21 11:57:09,998 DEBUG [StoreCloser-TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/e563dcb887eb411080fb5ce59692f5ff, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/69ddd8fc4c6c45078ad26db25a236568, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/a89eced45b54415a8c79345f21b0521f, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/66a5c69a5845493ca65368c66fa6dab4, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/3f5ef914f9294db6bdcf5fe357e413d9, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/3f27d1647db74eaeb4ad593602351780, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/6ec889323a7f4e049f84b9c9e3bca332, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/8a76553e26634920bf9f1a3455cd63eb, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/f3bcb07e030147e7bdab90dec2de9c99, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/44860811be204970af9357742aaead6d, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/cccf067d4c3f46da879c1cd2d5030d9c, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/a33871955a464f2ab68758d689706e38, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/684f34424bb245ffbdb0a6106699be42, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/aec612a2b8c943feb6f8f623b01737d1, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/229f87cf1f5248bfa2048d5f21ce1592, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/cb558ddc0c4c43e7bd5a417970368420, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/18b39d2506944108a539b603f6ecdc84, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/a88afc7dc59744b2a8975118df873fed, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/4a6359f049cd4b6c8318f659247cb349, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/8b68dfbaf7314b119eb7eb7ee1b87da4, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/f19a6b74f5d548e3acc85ada8affe5de, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/5f32f0c676994684acb666ac96ab9d71, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/1b5080d7ef444b328fa937fa566802a5, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/a17b925816d2443f888acac99c38dcf8, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/1ae9ddb1370b4d26bf263ec31a0eb180, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/80d691407a124458bd61a335298416db, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/244c8cc84a734307aebcf39a362bedf3, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/a77e8f47cca04c8f8c4908bae8dd3355, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/079cfcdf286f46338c2b24429f370e57, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/9e725d02022b48919e780ae768d95594, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/9e0cf8129de34d0aac51d8b1895b94e5, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/f97b4e45be594059a9f5d12a31b350db, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/1cf41e793aa945318a86580c2920f453, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/79b296a0b7894b4db082af2ec1460102] to archive 2023-03-21 11:57:09,999 DEBUG [StoreCloser-TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-21 11:57:10,000 DEBUG [StoreCloser-TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/e563dcb887eb411080fb5ce59692f5ff to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/e563dcb887eb411080fb5ce59692f5ff 2023-03-21 11:57:10,001 DEBUG [StoreCloser-TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/69ddd8fc4c6c45078ad26db25a236568 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/69ddd8fc4c6c45078ad26db25a236568 2023-03-21 11:57:10,002 DEBUG [StoreCloser-TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/a89eced45b54415a8c79345f21b0521f to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/a89eced45b54415a8c79345f21b0521f 2023-03-21 11:57:10,003 DEBUG [StoreCloser-TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/66a5c69a5845493ca65368c66fa6dab4 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/66a5c69a5845493ca65368c66fa6dab4 2023-03-21 11:57:10,004 DEBUG [StoreCloser-TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/3f5ef914f9294db6bdcf5fe357e413d9 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/3f5ef914f9294db6bdcf5fe357e413d9 2023-03-21 11:57:10,005 DEBUG [StoreCloser-TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/3f27d1647db74eaeb4ad593602351780 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/3f27d1647db74eaeb4ad593602351780 2023-03-21 11:57:10,006 DEBUG [StoreCloser-TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/6ec889323a7f4e049f84b9c9e3bca332 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/6ec889323a7f4e049f84b9c9e3bca332 2023-03-21 11:57:10,006 DEBUG [StoreCloser-TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/8a76553e26634920bf9f1a3455cd63eb to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/8a76553e26634920bf9f1a3455cd63eb 2023-03-21 11:57:10,007 DEBUG [StoreCloser-TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/f3bcb07e030147e7bdab90dec2de9c99 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/f3bcb07e030147e7bdab90dec2de9c99 2023-03-21 11:57:10,008 DEBUG [StoreCloser-TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/44860811be204970af9357742aaead6d to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/44860811be204970af9357742aaead6d 2023-03-21 11:57:10,009 DEBUG [StoreCloser-TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/cccf067d4c3f46da879c1cd2d5030d9c to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/cccf067d4c3f46da879c1cd2d5030d9c 2023-03-21 11:57:10,009 DEBUG [StoreCloser-TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/a33871955a464f2ab68758d689706e38 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/a33871955a464f2ab68758d689706e38 2023-03-21 11:57:10,010 DEBUG [StoreCloser-TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/684f34424bb245ffbdb0a6106699be42 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/684f34424bb245ffbdb0a6106699be42 2023-03-21 11:57:10,011 DEBUG [StoreCloser-TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/aec612a2b8c943feb6f8f623b01737d1 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/aec612a2b8c943feb6f8f623b01737d1 2023-03-21 11:57:10,012 DEBUG [StoreCloser-TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/229f87cf1f5248bfa2048d5f21ce1592 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/229f87cf1f5248bfa2048d5f21ce1592 2023-03-21 11:57:10,012 DEBUG [StoreCloser-TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/cb558ddc0c4c43e7bd5a417970368420 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/cb558ddc0c4c43e7bd5a417970368420 2023-03-21 11:57:10,013 DEBUG [StoreCloser-TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/18b39d2506944108a539b603f6ecdc84 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/18b39d2506944108a539b603f6ecdc84 2023-03-21 11:57:10,014 DEBUG [StoreCloser-TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/a88afc7dc59744b2a8975118df873fed to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/a88afc7dc59744b2a8975118df873fed 2023-03-21 11:57:10,015 DEBUG [StoreCloser-TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/4a6359f049cd4b6c8318f659247cb349 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/4a6359f049cd4b6c8318f659247cb349 2023-03-21 11:57:10,015 DEBUG [StoreCloser-TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/8b68dfbaf7314b119eb7eb7ee1b87da4 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/8b68dfbaf7314b119eb7eb7ee1b87da4 2023-03-21 11:57:10,016 DEBUG [StoreCloser-TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/f19a6b74f5d548e3acc85ada8affe5de to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/f19a6b74f5d548e3acc85ada8affe5de 2023-03-21 11:57:10,017 DEBUG [StoreCloser-TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/5f32f0c676994684acb666ac96ab9d71 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/5f32f0c676994684acb666ac96ab9d71 2023-03-21 11:57:10,018 DEBUG [StoreCloser-TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/1b5080d7ef444b328fa937fa566802a5 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/1b5080d7ef444b328fa937fa566802a5 2023-03-21 11:57:10,018 DEBUG [StoreCloser-TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/a17b925816d2443f888acac99c38dcf8 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/a17b925816d2443f888acac99c38dcf8 2023-03-21 11:57:10,019 DEBUG [StoreCloser-TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/1ae9ddb1370b4d26bf263ec31a0eb180 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/1ae9ddb1370b4d26bf263ec31a0eb180 2023-03-21 11:57:10,020 DEBUG [StoreCloser-TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/80d691407a124458bd61a335298416db to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/80d691407a124458bd61a335298416db 2023-03-21 11:57:10,021 DEBUG [StoreCloser-TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/244c8cc84a734307aebcf39a362bedf3 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/244c8cc84a734307aebcf39a362bedf3 2023-03-21 11:57:10,022 DEBUG [StoreCloser-TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/a77e8f47cca04c8f8c4908bae8dd3355 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/a77e8f47cca04c8f8c4908bae8dd3355 2023-03-21 11:57:10,022 DEBUG [StoreCloser-TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/079cfcdf286f46338c2b24429f370e57 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/079cfcdf286f46338c2b24429f370e57 2023-03-21 11:57:10,023 DEBUG [StoreCloser-TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/9e725d02022b48919e780ae768d95594 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/9e725d02022b48919e780ae768d95594 2023-03-21 11:57:10,024 DEBUG [StoreCloser-TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/9e0cf8129de34d0aac51d8b1895b94e5 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/9e0cf8129de34d0aac51d8b1895b94e5 2023-03-21 11:57:10,025 DEBUG [StoreCloser-TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/f97b4e45be594059a9f5d12a31b350db to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/f97b4e45be594059a9f5d12a31b350db 2023-03-21 11:57:10,026 DEBUG [StoreCloser-TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/1cf41e793aa945318a86580c2920f453 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/1cf41e793aa945318a86580c2920f453 2023-03-21 11:57:10,027 DEBUG [StoreCloser-TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/79b296a0b7894b4db082af2ec1460102 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/79b296a0b7894b4db082af2ec1460102 2023-03-21 11:57:10,030 DEBUG [StoreCloser-TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/56296638322246d2b38dc30711b72646, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/32882af9b9af404ea0e29eba90b740f5, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/3f862c593eca4303a374927588a54842, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/68df7fa6a7ce4bb48904c30c851a31c1, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/1a406e465c1f45cc92aa7bba0a4a1ca1, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/408f9df01f7b4a3eb85133064a4f9274, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/69d16325d4584f32803a73a8d8966126, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/495bd7317a7a4ab181d52c9dc37869da, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/96d379d14ae74735b980252c1771801f, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/3402724f53234b02b7a9a0cf51f14bdf, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/5fe1ce0ee1794bbfb9b089dcad916b9b, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/e9a9f4efe6d5480b89a71c01a862c614, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/9648bf22de7046fb9f7cadd8ec1af102, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/53d307cbe7f1425d8bc096bf778af22e, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/c5a054eebc3f4c09b4074604facab818, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/12fea09f3a5f4291902c1fbc5262beda, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/5506afddf73447acaa87276aaf0cb65e, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/7cf63990500243baa09700927d3c0e40, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/556f41fecfaa4409b53d38e3693ffef4, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/81fe49d653d0495c8c62d4419197eef6, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/1a78dcc7492b4daf89506206e57a5d20, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/7e8e573d142943a1b76db7e9a62ee270, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/2933409c7e814eb687ac0f3336f0d780, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/c4e09e3793bd455bab52bd48d7cdeafc, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/aefe42cada67458d9522aacfe7037a42, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/09944657c93748f8a3cbbce23a1ddb8d, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/953cdf5eb2864dd0b7f130d8c0a41278, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/2d7b8088b347457aa870f1e40cc1b28e, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/dae63057e46344bebf614d7df3f57ea4, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/699695417a1a4395a245bdf913962548, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/7b30d880e9e74a0388ee5e3275f7b2fe, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/4c31707dbddf4ba0a50b8b41a1afedbb, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/5781b45f8dab41ef8bff9369a1f3bcbc, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/b67b11fbc0bf45f9b9d6a81bd6bf7cec] to archive 2023-03-21 11:57:10,030 DEBUG [StoreCloser-TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-21 11:57:10,031 DEBUG [StoreCloser-TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/56296638322246d2b38dc30711b72646 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/56296638322246d2b38dc30711b72646 2023-03-21 11:57:10,032 DEBUG [StoreCloser-TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/32882af9b9af404ea0e29eba90b740f5 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/32882af9b9af404ea0e29eba90b740f5 2023-03-21 11:57:10,033 DEBUG [StoreCloser-TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/3f862c593eca4303a374927588a54842 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/3f862c593eca4303a374927588a54842 2023-03-21 11:57:10,034 DEBUG [StoreCloser-TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/68df7fa6a7ce4bb48904c30c851a31c1 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/68df7fa6a7ce4bb48904c30c851a31c1 2023-03-21 11:57:10,035 DEBUG [StoreCloser-TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/1a406e465c1f45cc92aa7bba0a4a1ca1 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/1a406e465c1f45cc92aa7bba0a4a1ca1 2023-03-21 11:57:10,036 DEBUG [StoreCloser-TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/408f9df01f7b4a3eb85133064a4f9274 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/408f9df01f7b4a3eb85133064a4f9274 2023-03-21 11:57:10,036 DEBUG [StoreCloser-TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/69d16325d4584f32803a73a8d8966126 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/69d16325d4584f32803a73a8d8966126 2023-03-21 11:57:10,037 DEBUG [StoreCloser-TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/495bd7317a7a4ab181d52c9dc37869da to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/495bd7317a7a4ab181d52c9dc37869da 2023-03-21 11:57:10,038 DEBUG [StoreCloser-TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/96d379d14ae74735b980252c1771801f to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/96d379d14ae74735b980252c1771801f 2023-03-21 11:57:10,038 DEBUG [StoreCloser-TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/3402724f53234b02b7a9a0cf51f14bdf to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/3402724f53234b02b7a9a0cf51f14bdf 2023-03-21 11:57:10,039 DEBUG [StoreCloser-TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/5fe1ce0ee1794bbfb9b089dcad916b9b to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/5fe1ce0ee1794bbfb9b089dcad916b9b 2023-03-21 11:57:10,040 DEBUG [StoreCloser-TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/e9a9f4efe6d5480b89a71c01a862c614 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/e9a9f4efe6d5480b89a71c01a862c614 2023-03-21 11:57:10,040 DEBUG [StoreCloser-TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/9648bf22de7046fb9f7cadd8ec1af102 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/9648bf22de7046fb9f7cadd8ec1af102 2023-03-21 11:57:10,041 DEBUG [StoreCloser-TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/53d307cbe7f1425d8bc096bf778af22e to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/53d307cbe7f1425d8bc096bf778af22e 2023-03-21 11:57:10,042 DEBUG [StoreCloser-TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/c5a054eebc3f4c09b4074604facab818 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/c5a054eebc3f4c09b4074604facab818 2023-03-21 11:57:10,042 DEBUG [StoreCloser-TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/12fea09f3a5f4291902c1fbc5262beda to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/12fea09f3a5f4291902c1fbc5262beda 2023-03-21 11:57:10,043 DEBUG [StoreCloser-TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/5506afddf73447acaa87276aaf0cb65e to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/5506afddf73447acaa87276aaf0cb65e 2023-03-21 11:57:10,044 DEBUG [StoreCloser-TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/7cf63990500243baa09700927d3c0e40 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/7cf63990500243baa09700927d3c0e40 2023-03-21 11:57:10,044 DEBUG [StoreCloser-TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/556f41fecfaa4409b53d38e3693ffef4 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/556f41fecfaa4409b53d38e3693ffef4 2023-03-21 11:57:10,045 DEBUG [StoreCloser-TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/81fe49d653d0495c8c62d4419197eef6 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/81fe49d653d0495c8c62d4419197eef6 2023-03-21 11:57:10,046 DEBUG [StoreCloser-TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/1a78dcc7492b4daf89506206e57a5d20 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/1a78dcc7492b4daf89506206e57a5d20 2023-03-21 11:57:10,047 DEBUG [StoreCloser-TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/7e8e573d142943a1b76db7e9a62ee270 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/7e8e573d142943a1b76db7e9a62ee270 2023-03-21 11:57:10,047 DEBUG [StoreCloser-TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/2933409c7e814eb687ac0f3336f0d780 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/2933409c7e814eb687ac0f3336f0d780 2023-03-21 11:57:10,048 DEBUG [StoreCloser-TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/c4e09e3793bd455bab52bd48d7cdeafc to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/c4e09e3793bd455bab52bd48d7cdeafc 2023-03-21 11:57:10,049 DEBUG [StoreCloser-TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/aefe42cada67458d9522aacfe7037a42 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/aefe42cada67458d9522aacfe7037a42 2023-03-21 11:57:10,050 DEBUG [StoreCloser-TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/09944657c93748f8a3cbbce23a1ddb8d to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/09944657c93748f8a3cbbce23a1ddb8d 2023-03-21 11:57:10,051 DEBUG [StoreCloser-TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/953cdf5eb2864dd0b7f130d8c0a41278 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/953cdf5eb2864dd0b7f130d8c0a41278 2023-03-21 11:57:10,052 DEBUG [StoreCloser-TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/2d7b8088b347457aa870f1e40cc1b28e to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/2d7b8088b347457aa870f1e40cc1b28e 2023-03-21 11:57:10,052 DEBUG [StoreCloser-TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/dae63057e46344bebf614d7df3f57ea4 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/dae63057e46344bebf614d7df3f57ea4 2023-03-21 11:57:10,053 DEBUG [StoreCloser-TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/699695417a1a4395a245bdf913962548 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/699695417a1a4395a245bdf913962548 2023-03-21 11:57:10,054 DEBUG [StoreCloser-TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/7b30d880e9e74a0388ee5e3275f7b2fe to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/7b30d880e9e74a0388ee5e3275f7b2fe 2023-03-21 11:57:10,055 DEBUG [StoreCloser-TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/4c31707dbddf4ba0a50b8b41a1afedbb to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/4c31707dbddf4ba0a50b8b41a1afedbb 2023-03-21 11:57:10,055 DEBUG [StoreCloser-TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/5781b45f8dab41ef8bff9369a1f3bcbc to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/5781b45f8dab41ef8bff9369a1f3bcbc 2023-03-21 11:57:10,056 DEBUG [StoreCloser-TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/b67b11fbc0bf45f9b9d6a81bd6bf7cec to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/b67b11fbc0bf45f9b9d6a81bd6bf7cec 2023-03-21 11:57:10,059 DEBUG [StoreCloser-TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/c7b344cea8784e15b7d70f21e807f05a, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/ff9c1d504a974bb3a8aba60a10dbc165, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/b2e74ca628444ad3b71837fe5e11bf80, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/152b222df2da49f0a6d33b99bbb9b965, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/dc3cd8f6322840d8a4c5637002ae7d47, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/2a60838532be43a38197e768a46b94c4, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/bebb5c9107844c1ab9855109e95ab265, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/fb93bed62d1c42d1bc570f64105a3573, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/1ded539d50604dc4b76062f211fdc006, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/b1aec5bb7dba4bc6a916c08828fc61c1, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/a4d6c61fc44b498b9ba4c6870a1fae71, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/6e2232a205484fa7a1e4811e81f53be5, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/86e07301b2664f2997d07456b9227f8c, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/1838cea787a5435ea08239055ab37b20, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/04ba3c1a363349fd9647273de76344c3, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/7ef9fe3da1e64dd3a63be4130e70ef27, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/2283fe3ccae44867ae8b2519e280987e, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/c57a4dc71aaf4f67aabfc9bb90c488c6, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/49532163ebb5459c885721be3443a74d, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/ca53b302ab324123804b4decf61921b7, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/b798803d08ba461c910d60a95eb91e95, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/1cabcceb8eba484095605a4db3b5ac7f, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/d96a44630578478fbdc148c147373a97, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/14c19255426741d98459554ed87ffc70, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/5865371a0afd4e39ace91c13292aaa5a, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/3bbcf6077c8647cb91b582719621334f, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/e95a7ba93afe4ab8b0b99e1160522c89, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/ab0b80841a3d437dafed450b71ad4b71, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/ef77dea4a1784839b14702320b3f3690, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/3d9d5993dcf4448aaa9fc36609b77c7d, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/79184bdb6a6a441c916b8406c42aa45f, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/ec76054a69ab4638910f08b314e01ccf, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/a9b7a83dd4284862929a431fa24f8411, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/84df4756430f42cfa1bdec9ea224c1bc] to archive 2023-03-21 11:57:10,060 DEBUG [StoreCloser-TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-21 11:57:10,061 DEBUG [StoreCloser-TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/c7b344cea8784e15b7d70f21e807f05a to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/c7b344cea8784e15b7d70f21e807f05a 2023-03-21 11:57:10,061 DEBUG [StoreCloser-TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/ff9c1d504a974bb3a8aba60a10dbc165 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/ff9c1d504a974bb3a8aba60a10dbc165 2023-03-21 11:57:10,062 DEBUG [StoreCloser-TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/b2e74ca628444ad3b71837fe5e11bf80 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/b2e74ca628444ad3b71837fe5e11bf80 2023-03-21 11:57:10,063 DEBUG [StoreCloser-TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/152b222df2da49f0a6d33b99bbb9b965 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/152b222df2da49f0a6d33b99bbb9b965 2023-03-21 11:57:10,064 DEBUG [StoreCloser-TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/dc3cd8f6322840d8a4c5637002ae7d47 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/dc3cd8f6322840d8a4c5637002ae7d47 2023-03-21 11:57:10,064 DEBUG [StoreCloser-TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/2a60838532be43a38197e768a46b94c4 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/2a60838532be43a38197e768a46b94c4 2023-03-21 11:57:10,065 DEBUG [StoreCloser-TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/bebb5c9107844c1ab9855109e95ab265 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/bebb5c9107844c1ab9855109e95ab265 2023-03-21 11:57:10,066 DEBUG [StoreCloser-TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/fb93bed62d1c42d1bc570f64105a3573 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/fb93bed62d1c42d1bc570f64105a3573 2023-03-21 11:57:10,066 DEBUG [StoreCloser-TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/1ded539d50604dc4b76062f211fdc006 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/1ded539d50604dc4b76062f211fdc006 2023-03-21 11:57:10,067 DEBUG [StoreCloser-TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/b1aec5bb7dba4bc6a916c08828fc61c1 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/b1aec5bb7dba4bc6a916c08828fc61c1 2023-03-21 11:57:10,068 DEBUG [StoreCloser-TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/a4d6c61fc44b498b9ba4c6870a1fae71 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/a4d6c61fc44b498b9ba4c6870a1fae71 2023-03-21 11:57:10,070 DEBUG [StoreCloser-TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/6e2232a205484fa7a1e4811e81f53be5 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/6e2232a205484fa7a1e4811e81f53be5 2023-03-21 11:57:10,070 DEBUG [StoreCloser-TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/86e07301b2664f2997d07456b9227f8c to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/86e07301b2664f2997d07456b9227f8c 2023-03-21 11:57:10,071 DEBUG [StoreCloser-TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/1838cea787a5435ea08239055ab37b20 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/1838cea787a5435ea08239055ab37b20 2023-03-21 11:57:10,072 DEBUG [StoreCloser-TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/04ba3c1a363349fd9647273de76344c3 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/04ba3c1a363349fd9647273de76344c3 2023-03-21 11:57:10,072 DEBUG [StoreCloser-TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/7ef9fe3da1e64dd3a63be4130e70ef27 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/7ef9fe3da1e64dd3a63be4130e70ef27 2023-03-21 11:57:10,073 DEBUG [StoreCloser-TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/2283fe3ccae44867ae8b2519e280987e to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/2283fe3ccae44867ae8b2519e280987e 2023-03-21 11:57:10,074 DEBUG [StoreCloser-TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/c57a4dc71aaf4f67aabfc9bb90c488c6 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/c57a4dc71aaf4f67aabfc9bb90c488c6 2023-03-21 11:57:10,074 DEBUG [StoreCloser-TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/49532163ebb5459c885721be3443a74d to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/49532163ebb5459c885721be3443a74d 2023-03-21 11:57:10,075 DEBUG [StoreCloser-TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/ca53b302ab324123804b4decf61921b7 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/ca53b302ab324123804b4decf61921b7 2023-03-21 11:57:10,076 DEBUG [StoreCloser-TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/b798803d08ba461c910d60a95eb91e95 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/b798803d08ba461c910d60a95eb91e95 2023-03-21 11:57:10,076 DEBUG [StoreCloser-TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/1cabcceb8eba484095605a4db3b5ac7f to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/1cabcceb8eba484095605a4db3b5ac7f 2023-03-21 11:57:10,077 DEBUG [StoreCloser-TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/d96a44630578478fbdc148c147373a97 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/d96a44630578478fbdc148c147373a97 2023-03-21 11:57:10,078 DEBUG [StoreCloser-TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/14c19255426741d98459554ed87ffc70 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/14c19255426741d98459554ed87ffc70 2023-03-21 11:57:10,078 DEBUG [StoreCloser-TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/5865371a0afd4e39ace91c13292aaa5a to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/5865371a0afd4e39ace91c13292aaa5a 2023-03-21 11:57:10,079 DEBUG [StoreCloser-TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/3bbcf6077c8647cb91b582719621334f to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/3bbcf6077c8647cb91b582719621334f 2023-03-21 11:57:10,080 DEBUG [StoreCloser-TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/e95a7ba93afe4ab8b0b99e1160522c89 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/e95a7ba93afe4ab8b0b99e1160522c89 2023-03-21 11:57:10,080 DEBUG [StoreCloser-TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/ab0b80841a3d437dafed450b71ad4b71 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/ab0b80841a3d437dafed450b71ad4b71 2023-03-21 11:57:10,081 DEBUG [StoreCloser-TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/ef77dea4a1784839b14702320b3f3690 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/ef77dea4a1784839b14702320b3f3690 2023-03-21 11:57:10,082 DEBUG [StoreCloser-TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/3d9d5993dcf4448aaa9fc36609b77c7d to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/3d9d5993dcf4448aaa9fc36609b77c7d 2023-03-21 11:57:10,083 DEBUG [StoreCloser-TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/79184bdb6a6a441c916b8406c42aa45f to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/79184bdb6a6a441c916b8406c42aa45f 2023-03-21 11:57:10,083 DEBUG [StoreCloser-TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/ec76054a69ab4638910f08b314e01ccf to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/ec76054a69ab4638910f08b314e01ccf 2023-03-21 11:57:10,084 DEBUG [StoreCloser-TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/a9b7a83dd4284862929a431fa24f8411 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/a9b7a83dd4284862929a431fa24f8411 2023-03-21 11:57:10,085 DEBUG [StoreCloser-TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/84df4756430f42cfa1bdec9ea224c1bc to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/84df4756430f42cfa1bdec9ea224c1bc 2023-03-21 11:57:10,089 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/recovered.edits/899.seqid, newMaxSeqId=899, maxSeqId=1 2023-03-21 11:57:10,090 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b. 2023-03-21 11:57:10,090 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 886ee010948b04e81bf5ede3c083ff8b: 2023-03-21 11:57:10,091 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 886ee010948b04e81bf5ede3c083ff8b 2023-03-21 11:57:10,092 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=886ee010948b04e81bf5ede3c083ff8b, regionState=CLOSED 2023-03-21 11:57:10,092 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679399830092"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679399830092"}]},"ts":"1679399830092"} 2023-03-21 11:57:10,094 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=33, resume processing ppid=32 2023-03-21 11:57:10,094 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=33, ppid=32, state=SUCCESS; CloseRegionProcedure 886ee010948b04e81bf5ede3c083ff8b, server=jenkins-hbase20.apache.org,34369,1679399732030 in 1.5310 sec 2023-03-21 11:57:10,095 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=32, resume processing ppid=31 2023-03-21 11:57:10,095 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=32, ppid=31, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=886ee010948b04e81bf5ede3c083ff8b, UNASSIGN in 1.5340 sec 2023-03-21 11:57:10,096 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679399830096"}]},"ts":"1679399830096"} 2023-03-21 11:57:10,096 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-03-21 11:57:10,097 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-03-21 11:57:10,098 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=31, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.5430 sec 2023-03-21 11:57:10,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-03-21 11:57:10,666 INFO [Listener at localhost.localdomain/37475] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 31 completed 2023-03-21 11:57:10,667 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.HMaster$5(2228): Client=jenkins//148.251.75.209 delete TestAcidGuarantees 2023-03-21 11:57:10,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure2.ProcedureExecutor(1029): Stored pid=34, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-03-21 11:57:10,671 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=34, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-21 11:57:10,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1227): Checking to see if procedure is done pid=34 2023-03-21 11:57:10,673 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=34, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-21 11:57:10,681 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/.tmp/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b 2023-03-21 11:57:10,684 DEBUG [HFileArchiver-7] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/.tmp/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A, FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/.tmp/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B, FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/.tmp/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C, FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/.tmp/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/recovered.edits] 2023-03-21 11:57:10,688 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/.tmp/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/1614fedec4ab41cb8dced9a8f44a8285 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/1614fedec4ab41cb8dced9a8f44a8285 2023-03-21 11:57:10,690 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/.tmp/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/72b0243279e047f9a9bf5744cf54c9ae to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/72b0243279e047f9a9bf5744cf54c9ae 2023-03-21 11:57:10,692 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/.tmp/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/e162bfc5c689471786861cb06ecb51fd to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/A/e162bfc5c689471786861cb06ecb51fd 2023-03-21 11:57:10,695 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/.tmp/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/355d56e4f1e7467292a6de61768a2cf6 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/355d56e4f1e7467292a6de61768a2cf6 2023-03-21 11:57:10,696 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/.tmp/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/c74e27680f30448abc03fb71786ec292 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/c74e27680f30448abc03fb71786ec292 2023-03-21 11:57:10,698 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/.tmp/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/fe5a0f28fe534719ae86613f54fb24cf to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/B/fe5a0f28fe534719ae86613f54fb24cf 2023-03-21 11:57:10,701 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/.tmp/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/14fd3fc9d69244e1813a7dd87dc799ab to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/14fd3fc9d69244e1813a7dd87dc799ab 2023-03-21 11:57:10,704 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/.tmp/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/a8cc952ac7e849168507119a6dcc6e34 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/a8cc952ac7e849168507119a6dcc6e34 2023-03-21 11:57:10,706 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/.tmp/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/dcd528dbdb03425f996d2cd53feb199d to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/C/dcd528dbdb03425f996d2cd53feb199d 2023-03-21 11:57:10,709 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/.tmp/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/recovered.edits/899.seqid to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b/recovered.edits/899.seqid 2023-03-21 11:57:10,710 DEBUG [HFileArchiver-7] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/.tmp/data/default/TestAcidGuarantees/886ee010948b04e81bf5ede3c083ff8b 2023-03-21 11:57:10,710 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-21 11:57:10,712 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=34, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-21 11:57:10,714 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-03-21 11:57:10,715 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-03-21 11:57:10,716 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=34, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-21 11:57:10,717 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-03-21 11:57:10,717 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1679399830717"}]},"ts":"9223372036854775807"} 2023-03-21 11:57:10,718 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-03-21 11:57:10,718 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 886ee010948b04e81bf5ede3c083ff8b, NAME => 'TestAcidGuarantees,,1679399800184.886ee010948b04e81bf5ede3c083ff8b.', STARTKEY => '', ENDKEY => ''}] 2023-03-21 11:57:10,718 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-03-21 11:57:10,719 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1679399830718"}]},"ts":"9223372036854775807"} 2023-03-21 11:57:10,720 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-03-21 11:57:10,721 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=34, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-21 11:57:10,721 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=34, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 54 msec 2023-03-21 11:57:10,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1227): Checking to see if procedure is done pid=34 2023-03-21 11:57:10,774 INFO [Listener at localhost.localdomain/37475] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 34 completed 2023-03-21 11:57:10,788 INFO [Listener at localhost.localdomain/37475] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testGetAtomicity Thread=224 (was 224), OpenFileDescriptor=556 (was 517) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=315 (was 307) - SystemLoadAverage LEAK? -, ProcessCount=167 (was 167), AvailableMemoryMB=8971 (was 9073) 2023-03-21 11:57:10,797 INFO [Listener at localhost.localdomain/37475] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobScanAtomicity Thread=224, OpenFileDescriptor=556, MaxFileDescriptor=60000, SystemLoadAverage=315, ProcessCount=167, AvailableMemoryMB=8971 2023-03-21 11:57:10,798 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-21 11:57:10,799 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.HMaster$4(2112): Client=jenkins//148.251.75.209 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-21 11:57:10,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure2.ProcedureExecutor(1029): Stored pid=35, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-03-21 11:57:10,801 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-03-21 11:57:10,801 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(697): Client=jenkins//148.251.75.209 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 35 2023-03-21 11:57:10,802 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-03-21 11:57:10,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-03-21 11:57:10,803 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/.tmp/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f 2023-03-21 11:57:10,804 DEBUG [HFileArchiver-8] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/.tmp/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f empty. 2023-03-21 11:57:10,804 DEBUG [HFileArchiver-8] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/.tmp/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f 2023-03-21 11:57:10,804 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-21 11:57:10,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-03-21 11:57:11,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-03-21 11:57:11,216 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-21 11:57:11,219 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => c7c660a43107606cd80c805b0450f72f, NAME => 'TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/.tmp 2023-03-21 11:57:11,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-03-21 11:57:11,628 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-21 11:57:11,628 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing c7c660a43107606cd80c805b0450f72f, disabling compactions & flushes 2023-03-21 11:57:11,628 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. 2023-03-21 11:57:11,628 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. 2023-03-21 11:57:11,628 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. after waiting 0 ms 2023-03-21 11:57:11,628 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. 2023-03-21 11:57:11,628 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. 2023-03-21 11:57:11,629 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for c7c660a43107606cd80c805b0450f72f: 2023-03-21 11:57:11,634 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-03-21 11:57:11,636 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679399831635"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679399831635"}]},"ts":"1679399831635"} 2023-03-21 11:57:11,637 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-03-21 11:57:11,638 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-03-21 11:57:11,638 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679399831638"}]},"ts":"1679399831638"} 2023-03-21 11:57:11,639 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-03-21 11:57:11,642 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=c7c660a43107606cd80c805b0450f72f, ASSIGN}] 2023-03-21 11:57:11,644 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=c7c660a43107606cd80c805b0450f72f, ASSIGN 2023-03-21 11:57:11,645 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=c7c660a43107606cd80c805b0450f72f, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,34369,1679399732030; forceNewPlan=false, retain=false 2023-03-21 11:57:11,797 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=c7c660a43107606cd80c805b0450f72f, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:11,797 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679399831797"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1679399831797"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679399831797"}]},"ts":"1679399831797"} 2023-03-21 11:57:11,799 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=37, ppid=36, state=RUNNABLE; OpenRegionProcedure c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030}] 2023-03-21 11:57:11,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-03-21 11:57:11,959 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. 2023-03-21 11:57:11,959 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => c7c660a43107606cd80c805b0450f72f, NAME => 'TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f.', STARTKEY => '', ENDKEY => ''} 2023-03-21 11:57:11,960 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees c7c660a43107606cd80c805b0450f72f 2023-03-21 11:57:11,960 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-21 11:57:11,960 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for c7c660a43107606cd80c805b0450f72f 2023-03-21 11:57:11,960 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for c7c660a43107606cd80c805b0450f72f 2023-03-21 11:57:11,962 INFO [StoreOpener-c7c660a43107606cd80c805b0450f72f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region c7c660a43107606cd80c805b0450f72f 2023-03-21 11:57:11,965 DEBUG [StoreOpener-c7c660a43107606cd80c805b0450f72f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/A 2023-03-21 11:57:11,965 DEBUG [StoreOpener-c7c660a43107606cd80c805b0450f72f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/A 2023-03-21 11:57:11,965 INFO [StoreOpener-c7c660a43107606cd80c805b0450f72f-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-21 11:57:11,965 INFO [StoreOpener-c7c660a43107606cd80c805b0450f72f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c7c660a43107606cd80c805b0450f72f columnFamilyName A 2023-03-21 11:57:11,966 INFO [StoreOpener-c7c660a43107606cd80c805b0450f72f-1] regionserver.HStore(310): Store=c7c660a43107606cd80c805b0450f72f/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-21 11:57:11,966 INFO [StoreOpener-c7c660a43107606cd80c805b0450f72f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region c7c660a43107606cd80c805b0450f72f 2023-03-21 11:57:11,968 DEBUG [StoreOpener-c7c660a43107606cd80c805b0450f72f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/B 2023-03-21 11:57:11,968 DEBUG [StoreOpener-c7c660a43107606cd80c805b0450f72f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/B 2023-03-21 11:57:11,968 INFO [StoreOpener-c7c660a43107606cd80c805b0450f72f-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-21 11:57:11,968 INFO [StoreOpener-c7c660a43107606cd80c805b0450f72f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c7c660a43107606cd80c805b0450f72f columnFamilyName B 2023-03-21 11:57:11,969 INFO [StoreOpener-c7c660a43107606cd80c805b0450f72f-1] regionserver.HStore(310): Store=c7c660a43107606cd80c805b0450f72f/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-21 11:57:11,969 INFO [StoreOpener-c7c660a43107606cd80c805b0450f72f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region c7c660a43107606cd80c805b0450f72f 2023-03-21 11:57:11,971 DEBUG [StoreOpener-c7c660a43107606cd80c805b0450f72f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/C 2023-03-21 11:57:11,971 DEBUG [StoreOpener-c7c660a43107606cd80c805b0450f72f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/C 2023-03-21 11:57:11,971 INFO [StoreOpener-c7c660a43107606cd80c805b0450f72f-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-21 11:57:11,971 INFO [StoreOpener-c7c660a43107606cd80c805b0450f72f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c7c660a43107606cd80c805b0450f72f columnFamilyName C 2023-03-21 11:57:11,972 INFO [StoreOpener-c7c660a43107606cd80c805b0450f72f-1] regionserver.HStore(310): Store=c7c660a43107606cd80c805b0450f72f/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-21 11:57:11,972 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. 2023-03-21 11:57:11,973 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f 2023-03-21 11:57:11,974 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f 2023-03-21 11:57:11,977 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-21 11:57:11,979 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for c7c660a43107606cd80c805b0450f72f 2023-03-21 11:57:11,982 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-21 11:57:11,983 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened c7c660a43107606cd80c805b0450f72f; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69771708, jitterRate=0.03967946767807007}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-21 11:57:11,983 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for c7c660a43107606cd80c805b0450f72f: 2023-03-21 11:57:11,984 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f., pid=37, masterSystemTime=1679399831951 2023-03-21 11:57:11,986 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. 2023-03-21 11:57:11,986 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. 2023-03-21 11:57:11,986 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=c7c660a43107606cd80c805b0450f72f, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:11,986 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679399831986"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1679399831986"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1679399831986"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1679399831986"}]},"ts":"1679399831986"} 2023-03-21 11:57:11,989 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=37, resume processing ppid=36 2023-03-21 11:57:11,989 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=37, ppid=36, state=SUCCESS; OpenRegionProcedure c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 in 189 msec 2023-03-21 11:57:11,991 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=36, resume processing ppid=35 2023-03-21 11:57:11,991 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=36, ppid=35, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=c7c660a43107606cd80c805b0450f72f, ASSIGN in 347 msec 2023-03-21 11:57:11,992 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-03-21 11:57:11,992 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679399831992"}]},"ts":"1679399831992"} 2023-03-21 11:57:11,993 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-03-21 11:57:11,995 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-03-21 11:57:11,996 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=35, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1960 sec 2023-03-21 11:57:12,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-03-21 11:57:12,912 INFO [Listener at localhost.localdomain/37475] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 35 completed 2023-03-21 11:57:12,913 DEBUG [Listener at localhost.localdomain/37475] zookeeper.ReadOnlyZKClient(139): Connect 0x648e005e to 127.0.0.1:61761 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-21 11:57:12,921 DEBUG [Listener at localhost.localdomain/37475] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3a8219c3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-21 11:57:12,923 DEBUG [Listener at localhost.localdomain/37475] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-21 11:57:12,926 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:40976, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-21 11:57:12,927 DEBUG [Listener at localhost.localdomain/37475] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-03-21 11:57:12,927 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:49190, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-03-21 11:57:12,929 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-21 11:57:12,929 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.HMaster$12(2454): Client=jenkins//148.251.75.209 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-21 11:57:12,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure2.ProcedureExecutor(1029): Stored pid=38, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-03-21 11:57:13,339 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-03-21 11:57:13,341 DEBUG [PEWorker-4] util.FSTableDescriptors(519): Deleted hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-21 11:57:13,341 INFO [PEWorker-4] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-03-21 11:57:13,346 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=39, ppid=38, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-03-21 11:57:13,351 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=c7c660a43107606cd80c805b0450f72f, REOPEN/MOVE}] 2023-03-21 11:57:13,351 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=c7c660a43107606cd80c805b0450f72f, REOPEN/MOVE 2023-03-21 11:57:13,352 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=c7c660a43107606cd80c805b0450f72f, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:13,352 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679399833352"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1679399833352"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679399833352"}]},"ts":"1679399833352"} 2023-03-21 11:57:13,353 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=41, ppid=40, state=RUNNABLE; CloseRegionProcedure c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030}] 2023-03-21 11:57:13,506 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close c7c660a43107606cd80c805b0450f72f 2023-03-21 11:57:13,507 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing c7c660a43107606cd80c805b0450f72f, disabling compactions & flushes 2023-03-21 11:57:13,507 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. 2023-03-21 11:57:13,508 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. 2023-03-21 11:57:13,508 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. after waiting 0 ms 2023-03-21 11:57:13,508 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. 2023-03-21 11:57:13,515 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-03-21 11:57:13,516 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. 2023-03-21 11:57:13,516 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for c7c660a43107606cd80c805b0450f72f: 2023-03-21 11:57:13,516 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(3509): Not adding moved region record: c7c660a43107606cd80c805b0450f72f to self. 2023-03-21 11:57:13,518 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed c7c660a43107606cd80c805b0450f72f 2023-03-21 11:57:13,519 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=c7c660a43107606cd80c805b0450f72f, regionState=CLOSED 2023-03-21 11:57:13,519 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679399833519"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679399833519"}]},"ts":"1679399833519"} 2023-03-21 11:57:13,522 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=41, resume processing ppid=40 2023-03-21 11:57:13,522 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=41, ppid=40, state=SUCCESS; CloseRegionProcedure c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 in 167 msec 2023-03-21 11:57:13,523 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=c7c660a43107606cd80c805b0450f72f, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase20.apache.org,34369,1679399732030; forceNewPlan=false, retain=true 2023-03-21 11:57:13,674 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=c7c660a43107606cd80c805b0450f72f, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:13,674 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679399833673"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1679399833673"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679399833673"}]},"ts":"1679399833673"} 2023-03-21 11:57:13,677 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=42, ppid=40, state=RUNNABLE; OpenRegionProcedure c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030}] 2023-03-21 11:57:13,836 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. 2023-03-21 11:57:13,836 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => c7c660a43107606cd80c805b0450f72f, NAME => 'TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f.', STARTKEY => '', ENDKEY => ''} 2023-03-21 11:57:13,837 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees c7c660a43107606cd80c805b0450f72f 2023-03-21 11:57:13,837 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-21 11:57:13,838 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for c7c660a43107606cd80c805b0450f72f 2023-03-21 11:57:13,838 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for c7c660a43107606cd80c805b0450f72f 2023-03-21 11:57:13,839 INFO [StoreOpener-c7c660a43107606cd80c805b0450f72f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region c7c660a43107606cd80c805b0450f72f 2023-03-21 11:57:13,840 DEBUG [StoreOpener-c7c660a43107606cd80c805b0450f72f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/A 2023-03-21 11:57:13,840 DEBUG [StoreOpener-c7c660a43107606cd80c805b0450f72f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/A 2023-03-21 11:57:13,841 INFO [StoreOpener-c7c660a43107606cd80c805b0450f72f-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-21 11:57:13,841 INFO [StoreOpener-c7c660a43107606cd80c805b0450f72f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c7c660a43107606cd80c805b0450f72f columnFamilyName A 2023-03-21 11:57:13,843 INFO [StoreOpener-c7c660a43107606cd80c805b0450f72f-1] regionserver.HStore(310): Store=c7c660a43107606cd80c805b0450f72f/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-21 11:57:13,843 INFO [StoreOpener-c7c660a43107606cd80c805b0450f72f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region c7c660a43107606cd80c805b0450f72f 2023-03-21 11:57:13,844 DEBUG [StoreOpener-c7c660a43107606cd80c805b0450f72f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/B 2023-03-21 11:57:13,844 DEBUG [StoreOpener-c7c660a43107606cd80c805b0450f72f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/B 2023-03-21 11:57:13,844 INFO [StoreOpener-c7c660a43107606cd80c805b0450f72f-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-21 11:57:13,845 INFO [StoreOpener-c7c660a43107606cd80c805b0450f72f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c7c660a43107606cd80c805b0450f72f columnFamilyName B 2023-03-21 11:57:13,845 INFO [StoreOpener-c7c660a43107606cd80c805b0450f72f-1] regionserver.HStore(310): Store=c7c660a43107606cd80c805b0450f72f/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-21 11:57:13,846 INFO [StoreOpener-c7c660a43107606cd80c805b0450f72f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region c7c660a43107606cd80c805b0450f72f 2023-03-21 11:57:13,847 DEBUG [StoreOpener-c7c660a43107606cd80c805b0450f72f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/C 2023-03-21 11:57:13,847 DEBUG [StoreOpener-c7c660a43107606cd80c805b0450f72f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/C 2023-03-21 11:57:13,847 INFO [StoreOpener-c7c660a43107606cd80c805b0450f72f-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-21 11:57:13,847 INFO [StoreOpener-c7c660a43107606cd80c805b0450f72f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c7c660a43107606cd80c805b0450f72f columnFamilyName C 2023-03-21 11:57:13,848 INFO [StoreOpener-c7c660a43107606cd80c805b0450f72f-1] regionserver.HStore(310): Store=c7c660a43107606cd80c805b0450f72f/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-21 11:57:13,848 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. 2023-03-21 11:57:13,849 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f 2023-03-21 11:57:13,851 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f 2023-03-21 11:57:13,854 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-21 11:57:13,856 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for c7c660a43107606cd80c805b0450f72f 2023-03-21 11:57:13,857 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened c7c660a43107606cd80c805b0450f72f; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66321579, jitterRate=-0.011731460690498352}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-21 11:57:13,857 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for c7c660a43107606cd80c805b0450f72f: 2023-03-21 11:57:13,858 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f., pid=42, masterSystemTime=1679399833830 2023-03-21 11:57:13,860 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. 2023-03-21 11:57:13,860 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. 2023-03-21 11:57:13,860 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=c7c660a43107606cd80c805b0450f72f, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:13,860 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679399833860"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1679399833860"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1679399833860"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1679399833860"}]},"ts":"1679399833860"} 2023-03-21 11:57:13,863 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=42, resume processing ppid=40 2023-03-21 11:57:13,864 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=42, ppid=40, state=SUCCESS; OpenRegionProcedure c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 in 184 msec 2023-03-21 11:57:13,865 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=40, resume processing ppid=39 2023-03-21 11:57:13,866 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=40, ppid=39, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=c7c660a43107606cd80c805b0450f72f, REOPEN/MOVE in 513 msec 2023-03-21 11:57:13,868 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=39, resume processing ppid=38 2023-03-21 11:57:13,869 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=39, ppid=38, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 520 msec 2023-03-21 11:57:13,872 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=38, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 940 msec 2023-03-21 11:57:13,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1227): Checking to see if procedure is done pid=38 2023-03-21 11:57:13,874 DEBUG [Listener at localhost.localdomain/37475] zookeeper.ReadOnlyZKClient(139): Connect 0x3acdb9c4 to 127.0.0.1:61761 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-21 11:57:13,881 DEBUG [Listener at localhost.localdomain/37475] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2ed5d233, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-21 11:57:13,881 DEBUG [Listener at localhost.localdomain/37475] zookeeper.ReadOnlyZKClient(139): Connect 0x3d941bc8 to 127.0.0.1:61761 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-21 11:57:13,885 DEBUG [Listener at localhost.localdomain/37475] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6a409627, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-21 11:57:13,886 DEBUG [Listener at localhost.localdomain/37475] zookeeper.ReadOnlyZKClient(139): Connect 0x727ea8af to 127.0.0.1:61761 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-21 11:57:13,890 DEBUG [Listener at localhost.localdomain/37475] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6f29a773, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-21 11:57:13,890 DEBUG [Listener at localhost.localdomain/37475] zookeeper.ReadOnlyZKClient(139): Connect 0x4f54fc1a to 127.0.0.1:61761 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-21 11:57:13,895 DEBUG [Listener at localhost.localdomain/37475] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@48681e8f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-21 11:57:13,895 DEBUG [Listener at localhost.localdomain/37475] zookeeper.ReadOnlyZKClient(139): Connect 0x5c70a497 to 127.0.0.1:61761 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-21 11:57:13,900 DEBUG [Listener at localhost.localdomain/37475] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@342c272f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-21 11:57:13,901 DEBUG [Listener at localhost.localdomain/37475] zookeeper.ReadOnlyZKClient(139): Connect 0x4e90d228 to 127.0.0.1:61761 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-21 11:57:13,906 DEBUG [Listener at localhost.localdomain/37475] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1bd0918a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-21 11:57:13,907 DEBUG [Listener at localhost.localdomain/37475] zookeeper.ReadOnlyZKClient(139): Connect 0x1aec6811 to 127.0.0.1:61761 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-21 11:57:13,920 DEBUG [Listener at localhost.localdomain/37475] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5225f8f6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-21 11:57:13,921 DEBUG [Listener at localhost.localdomain/37475] zookeeper.ReadOnlyZKClient(139): Connect 0x6d951686 to 127.0.0.1:61761 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-21 11:57:13,928 DEBUG [Listener at localhost.localdomain/37475] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@144047c5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-21 11:57:13,929 DEBUG [Listener at localhost.localdomain/37475] zookeeper.ReadOnlyZKClient(139): Connect 0x17385c96 to 127.0.0.1:61761 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-21 11:57:13,933 DEBUG [Listener at localhost.localdomain/37475] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@525c59d1, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-21 11:57:13,933 DEBUG [Listener at localhost.localdomain/37475] zookeeper.ReadOnlyZKClient(139): Connect 0x52829d32 to 127.0.0.1:61761 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-21 11:57:13,937 DEBUG [Listener at localhost.localdomain/37475] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@78ba718, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-21 11:57:13,949 DEBUG [hconnection-0x2066735c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-21 11:57:13,950 DEBUG [hconnection-0x404ee2d4-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-21 11:57:13,950 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:40988, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-21 11:57:13,953 DEBUG [hconnection-0x6770f2f3-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-21 11:57:13,953 DEBUG [hconnection-0x1e0d3b2-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-21 11:57:13,953 DEBUG [hconnection-0x5dd69ee4-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-21 11:57:13,953 DEBUG [hconnection-0x8355ebd-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-21 11:57:13,956 DEBUG [hconnection-0x46d81a83-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-21 11:57:13,956 DEBUG [hconnection-0x5dabd0c3-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-21 11:57:13,956 DEBUG [hconnection-0x7737af63-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-21 11:57:13,957 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:41008, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-21 11:57:13,958 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:41010, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-21 11:57:13,958 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:41046, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-21 11:57:13,957 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:40994, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-21 11:57:13,959 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:41034, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-21 11:57:13,959 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:41056, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-21 11:57:13,960 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:41068, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-21 11:57:13,974 DEBUG [hconnection-0x557dc4e3-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-21 11:57:13,974 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:41084, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-21 11:57:13,978 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:41100, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-21 11:57:13,981 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:57:13,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(9158): Flush requested on c7c660a43107606cd80c805b0450f72f 2023-03-21 11:57:13,987 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c7c660a43107606cd80c805b0450f72f 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-21 11:57:13,988 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c7c660a43107606cd80c805b0450f72f, store=A 2023-03-21 11:57:13,988 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:57:13,988 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c7c660a43107606cd80c805b0450f72f, store=B 2023-03-21 11:57:13,988 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:57:13,988 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c7c660a43107606cd80c805b0450f72f, store=C 2023-03-21 11:57:13,988 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:57:13,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:57:13,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:57:13,999 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:57:13,999 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:13,999 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:57:13,999 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:57:14,002 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:14,002 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:14,005 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:14,005 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,006 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,006 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:57:14,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:14,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:14,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:14,007 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:14,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:57:14,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:14,008 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:57:14,010 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:57:14,010 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:57:14,010 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:57:14,010 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:14,010 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:57:14,012 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:14,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41084 deadline: 1679399894010, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,012 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:14,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41056 deadline: 1679399894011, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,012 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:14,013 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:14,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41010 deadline: 1679399894011, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41068 deadline: 1679399894012, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,013 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:14,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41034 deadline: 1679399894012, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,018 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:57:14,018 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. 2023-03-21 11:57:14,018 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:57:14,018 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. started... 2023-03-21 11:57:14,018 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. as already flushing 2023-03-21 11:57:14,018 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. 2023-03-21 11:57:14,018 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:57:14,018 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:57:14,018 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,025 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:57:14,025 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:57:14,027 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:14,027 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:14,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:14,027 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:14,027 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:57:14,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:14,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:14,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:14,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:14,028 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:57:14,028 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c61d3ac[Count = 0] remaining members to acquire global barrier 2023-03-21 11:57:14,028 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:57:14,028 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:14,029 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:14,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:14,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:14,029 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:57:14,029 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:57:14,029 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:57:14,030 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,030 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:57:14,030 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:14,030 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:57:14,031 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:57:14,031 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:57:14,031 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:57:14,031 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:57:14,031 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:57:14,031 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:57:14,031 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:57:14,032 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:14,033 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:14,033 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:14,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:14,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:14,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:14,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:14,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:14,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:14,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:14,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:14,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:14,034 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032119f5396649064f43a8a71f50d58a9349 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032119f5396649064f43a8a71f50d58a9349 2023-03-21 11:57:14,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:14,034 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:14,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:14,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:14,035 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:14,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,037 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,037 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:14,037 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:57:14,037 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:57:14,037 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:14,037 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:14,037 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,037 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:14,037 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:14,037 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:14,037 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:14,037 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:14,037 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:57:14,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:14,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:14,038 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:57:14,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:14,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:14,116 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:14,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41084 deadline: 1679399894113, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,119 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:14,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41010 deadline: 1679399894113, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,119 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:14,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41056 deadline: 1679399894113, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,120 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:14,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41068 deadline: 1679399894113, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,120 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:14,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41034 deadline: 1679399894114, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,138 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:14,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:14,140 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:57:14,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:57:14,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:57:14,146 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:57:14,146 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:14,146 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:57:14,146 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:57:14,147 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:14,147 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:14,148 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:14,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:14,148 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:14,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:14,148 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,148 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:57:14,148 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:14,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:57:14,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:14,149 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:57:14,154 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:57:14,154 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:57:14,154 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:57:14,154 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:14,154 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:57:14,158 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:57:14,158 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:57:14,158 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. 2023-03-21 11:57:14,158 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. started... 2023-03-21 11:57:14,158 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. as already flushing 2023-03-21 11:57:14,158 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. 2023-03-21 11:57:14,158 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:57:14,158 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:57:14,158 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,158 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:57:14,158 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:57:14,162 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,162 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:14,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:14,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:14,163 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:14,163 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:57:14,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:14,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:14,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:14,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:14,164 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:57:14,164 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@13e48ea7[Count = 0] remaining members to acquire global barrier 2023-03-21 11:57:14,164 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:57:14,164 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:14,164 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:14,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:14,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:14,164 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:57:14,164 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:57:14,164 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:57:14,164 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,165 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:57:14,165 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,165 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:57:14,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:14,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:14,165 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:14,165 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:57:14,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:14,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:14,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:14,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:14,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:14,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:57:14,167 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:57:14,167 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:57:14,167 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:57:14,167 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:57:14,167 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:57:14,167 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:57:14,168 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:14,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:14,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:14,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:14,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:14,169 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:14,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:14,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:14,169 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:14,170 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:14,170 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:14,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:14,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:14,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:14,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:14,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:14,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:14,171 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:14,171 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:14,171 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:57:14,171 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:57:14,171 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:57:14,171 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:14,171 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:57:14,171 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:57:14,171 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:57:14,171 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:14,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:14,171 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:14,171 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:14,171 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,171 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:14,171 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:14,171 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:14,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:14,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:14,271 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:14,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:14,273 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:57:14,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:57:14,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:57:14,275 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:57:14,275 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:14,275 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:57:14,275 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:57:14,275 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:14,275 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:14,276 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:14,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:14,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:14,276 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:14,276 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,276 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:57:14,276 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:14,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:57:14,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:14,276 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:57:14,278 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:57:14,278 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:57:14,278 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:57:14,278 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:14,278 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:57:14,286 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:57:14,286 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:57:14,286 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. 2023-03-21 11:57:14,286 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. started... 2023-03-21 11:57:14,286 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. as already flushing 2023-03-21 11:57:14,286 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. 2023-03-21 11:57:14,286 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:57:14,286 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:57:14,286 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,286 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:57:14,286 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:57:14,291 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:14,291 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:14,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:14,291 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:14,291 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:57:14,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:14,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:14,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:14,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:14,292 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:57:14,292 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4ac1766a[Count = 0] remaining members to acquire global barrier 2023-03-21 11:57:14,292 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:57:14,292 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:14,293 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:14,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:14,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:14,293 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:57:14,293 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:57:14,293 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:57:14,293 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,293 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:57:14,294 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:57:14,294 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:14,294 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:57:14,294 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:14,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:14,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:14,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:14,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:14,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:14,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:14,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:57:14,296 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:57:14,296 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:57:14,296 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:57:14,296 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:57:14,296 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:57:14,296 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:57:14,297 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:14,297 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:14,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:14,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:14,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:14,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:14,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:14,297 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:14,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:14,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:14,297 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:14,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:14,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:14,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:14,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:14,299 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:14,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:14,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,300 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,300 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:14,300 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:14,300 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,300 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:14,300 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:14,300 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:57:14,300 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:57:14,300 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:14,300 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:14,300 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:57:14,300 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:57:14,300 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:57:14,301 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:14,301 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:14,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:14,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:14,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:14,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:14,322 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:14,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41084 deadline: 1679399894319, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,323 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:14,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41034 deadline: 1679399894321, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,323 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:14,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41010 deadline: 1679399894321, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,324 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:14,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41068 deadline: 1679399894321, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,324 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:14,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41056 deadline: 1679399894322, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,400 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:14,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:14,402 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:57:14,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:57:14,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:57:14,403 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:57:14,403 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:14,403 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:57:14,403 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:57:14,404 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:14,404 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:14,404 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:14,404 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:14,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:14,405 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,405 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:57:14,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:14,405 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:14,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:57:14,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:14,405 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:57:14,405 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:57:14,405 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:57:14,405 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:57:14,405 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:14,405 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:57:14,406 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:57:14,406 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:57:14,406 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. 2023-03-21 11:57:14,406 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. started... 2023-03-21 11:57:14,406 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. as already flushing 2023-03-21 11:57:14,406 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. 2023-03-21 11:57:14,406 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:57:14,406 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:57:14,406 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,406 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:57:14,406 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:57:14,407 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:14,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:14,407 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:14,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:14,407 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:14,407 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:57:14,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:14,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:14,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:14,408 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:57:14,408 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ad846b[Count = 0] remaining members to acquire global barrier 2023-03-21 11:57:14,408 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:57:14,408 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:14,409 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:14,409 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,409 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:57:14,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:14,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:14,410 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:57:14,410 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:57:14,410 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:57:14,410 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:57:14,410 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:14,410 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,410 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:57:14,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:14,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:14,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:14,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:14,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:14,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:14,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:14,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:57:14,412 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:57:14,412 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:57:14,412 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:57:14,412 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:57:14,412 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:57:14,412 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:57:14,413 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:14,413 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:14,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:14,413 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:14,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:14,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:14,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:14,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:14,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:14,413 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:14,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:14,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:14,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:14,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:14,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:14,414 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:14,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:14,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,416 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,416 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:14,416 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:14,416 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,416 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:57:14,416 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:57:14,416 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:14,416 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:14,416 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:14,416 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:14,417 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:57:14,417 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:14,417 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:57:14,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:14,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:14,417 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:14,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:14,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:14,449 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=18, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/.tmp/A/70b70bb7cbf84daabcbc34626c25b38f 2023-03-21 11:57:14,517 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:14,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:14,518 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:57:14,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:57:14,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:57:14,520 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:57:14,520 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:14,520 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:57:14,520 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:57:14,520 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:14,520 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:14,521 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:14,521 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:14,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:14,521 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,521 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:57:14,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:14,521 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:14,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:57:14,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:14,521 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:57:14,522 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:57:14,522 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:57:14,522 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:57:14,522 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:14,522 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:57:14,522 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:57:14,522 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:57:14,522 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. 2023-03-21 11:57:14,522 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. started... 2023-03-21 11:57:14,522 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. as already flushing 2023-03-21 11:57:14,522 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. 2023-03-21 11:57:14,522 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:57:14,522 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:57:14,522 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,522 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:57:14,522 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:57:14,523 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,523 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:14,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:14,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:14,523 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:14,523 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:57:14,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:14,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:14,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:14,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:14,524 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:57:14,524 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:57:14,524 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7223cb61[Count = 0] remaining members to acquire global barrier 2023-03-21 11:57:14,524 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:14,525 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:14,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:14,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:14,525 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:57:14,525 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:57:14,525 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:57:14,525 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,525 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:57:14,526 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,526 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:57:14,526 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:14,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:14,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:14,526 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:57:14,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:14,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:14,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:14,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:14,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:14,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:57:14,527 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:57:14,527 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:57:14,528 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:57:14,528 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:57:14,528 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:57:14,528 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:57:14,528 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:14,528 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:14,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:14,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:14,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:14,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:14,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:14,528 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:14,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:14,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:14,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:14,529 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:14,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:14,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:14,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:14,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,531 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:14,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:14,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,532 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,532 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:14,532 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:14,532 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:14,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:14,532 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:14,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:14,532 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:57:14,532 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:57:14,532 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:14,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:14,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:14,532 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:57:14,532 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,533 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:57:14,533 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:14,533 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:57:14,533 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:14,533 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:14,626 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:14,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41084 deadline: 1679399894623, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,626 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:14,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41034 deadline: 1679399894624, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,627 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:14,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41056 deadline: 1679399894624, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,628 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:14,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41010 deadline: 1679399894625, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,628 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:14,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41068 deadline: 1679399894626, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,633 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:14,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:14,635 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:57:14,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:57:14,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:57:14,636 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:57:14,636 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:14,636 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:57:14,636 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:57:14,636 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:14,636 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:14,637 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,637 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:14,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:14,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:14,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:14,638 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,638 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:57:14,638 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:14,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:57:14,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:14,642 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:57:14,642 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:57:14,642 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:57:14,642 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:57:14,642 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:14,642 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:57:14,646 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:57:14,646 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:57:14,646 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. 2023-03-21 11:57:14,646 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. started... 2023-03-21 11:57:14,646 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. as already flushing 2023-03-21 11:57:14,646 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. 2023-03-21 11:57:14,646 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:57:14,646 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:57:14,646 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,646 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:57:14,646 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:57:14,653 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:14,653 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:14,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:14,653 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:14,653 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:57:14,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:14,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:14,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:14,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:14,654 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:57:14,654 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4505e234[Count = 0] remaining members to acquire global barrier 2023-03-21 11:57:14,654 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:57:14,654 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:14,657 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:14,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:14,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:14,657 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:57:14,657 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:57:14,657 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,658 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:57:14,658 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:57:14,658 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:57:14,658 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:14,658 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,658 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:57:14,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:14,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:14,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:14,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:14,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:14,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:14,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:14,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:57:14,660 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:57:14,660 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:57:14,660 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:57:14,660 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:57:14,660 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:57:14,660 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:57:14,661 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:14,661 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:14,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:14,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:14,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:14,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:14,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:14,661 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:14,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:14,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:14,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:14,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:14,661 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:14,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:14,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:14,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,662 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:14,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:14,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,665 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:14,665 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:14,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:14,665 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:14,665 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:14,665 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:14,665 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:57:14,665 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:57:14,665 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:14,665 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,665 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:14,665 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:14,665 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:14,665 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:57:14,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:14,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:14,665 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:57:14,765 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:14,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:14,767 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:57:14,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:57:14,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:57:14,768 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:57:14,768 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:14,768 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:57:14,768 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:57:14,768 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:14,768 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:14,769 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:14,769 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:14,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:14,769 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,769 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:57:14,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:14,769 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:14,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:57:14,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:14,770 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:57:14,770 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:57:14,770 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:57:14,770 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:14,770 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:57:14,770 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:57:14,770 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:57:14,770 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. 2023-03-21 11:57:14,770 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. started... 2023-03-21 11:57:14,770 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. as already flushing 2023-03-21 11:57:14,770 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. 2023-03-21 11:57:14,770 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:57:14,770 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:57:14,770 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,770 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:57:14,771 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:57:14,771 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:14,771 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:14,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:14,771 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:14,771 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:57:14,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:14,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:14,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:14,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:14,772 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:57:14,772 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b6aed8[Count = 0] remaining members to acquire global barrier 2023-03-21 11:57:14,772 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:57:14,772 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:14,773 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:14,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:14,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:14,773 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,773 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:57:14,773 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:57:14,773 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:57:14,773 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:57:14,774 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,774 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:57:14,774 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:14,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:14,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:14,774 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:57:14,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:14,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:14,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:14,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:14,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:14,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:57:14,776 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:57:14,776 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:57:14,776 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:57:14,776 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:57:14,776 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:57:14,776 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:57:14,777 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:14,777 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:14,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:14,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:14,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:14,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:14,777 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:14,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:14,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:14,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:14,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:14,778 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:14,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:14,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:14,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:14,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,779 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:14,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:14,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,781 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,781 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:14,781 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:14,781 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:14,781 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:14,781 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:14,781 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:57:14,781 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:57:14,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:14,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:14,781 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,781 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:14,781 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:14,781 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:57:14,781 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:14,781 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:57:14,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:14,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:14,858 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=18 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/.tmp/B/6d7e3a7a2c0f4a19864a119375c9124b 2023-03-21 11:57:14,867 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=18 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/.tmp/C/c7e173a2da7346f79762e26607b8698b 2023-03-21 11:57:14,870 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/.tmp/A/70b70bb7cbf84daabcbc34626c25b38f as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/A/70b70bb7cbf84daabcbc34626c25b38f 2023-03-21 11:57:14,873 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/A/70b70bb7cbf84daabcbc34626c25b38f, entries=150, sequenceid=18, filesize=25.1 K 2023-03-21 11:57:14,873 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/.tmp/B/6d7e3a7a2c0f4a19864a119375c9124b as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/B/6d7e3a7a2c0f4a19864a119375c9124b 2023-03-21 11:57:14,876 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/B/6d7e3a7a2c0f4a19864a119375c9124b, entries=150, sequenceid=18, filesize=11.6 K 2023-03-21 11:57:14,876 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/.tmp/C/c7e173a2da7346f79762e26607b8698b as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/C/c7e173a2da7346f79762e26607b8698b 2023-03-21 11:57:14,881 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/C/c7e173a2da7346f79762e26607b8698b, entries=150, sequenceid=18, filesize=11.6 K 2023-03-21 11:57:14,881 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:14,882 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for c7c660a43107606cd80c805b0450f72f in 895ms, sequenceid=18, compaction requested=false 2023-03-21 11:57:14,882 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c7c660a43107606cd80c805b0450f72f: 2023-03-21 11:57:14,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:14,883 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:57:14,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:57:14,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:57:14,884 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:57:14,884 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:14,884 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:57:14,884 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:57:14,884 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:14,884 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:14,885 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,885 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:14,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:14,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:14,885 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:14,885 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:57:14,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:14,885 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:14,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:57:14,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:14,886 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:57:14,886 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:57:14,886 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:57:14,886 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:57:14,886 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:14,886 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:57:14,886 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:57:14,886 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:57:14,886 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. 2023-03-21 11:57:14,886 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. started... 2023-03-21 11:57:14,886 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c7c660a43107606cd80c805b0450f72f 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-03-21 11:57:14,887 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c7c660a43107606cd80c805b0450f72f, store=A 2023-03-21 11:57:14,887 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:57:14,887 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c7c660a43107606cd80c805b0450f72f, store=B 2023-03-21 11:57:14,887 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:57:14,887 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c7c660a43107606cd80c805b0450f72f, store=C 2023-03-21 11:57:14,887 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:57:14,911 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230321201215bd17a14f3c93087705bd32bd97 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230321201215bd17a14f3c93087705bd32bd97 2023-03-21 11:57:15,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(9158): Flush requested on c7c660a43107606cd80c805b0450f72f 2023-03-21 11:57:15,130 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. as already flushing 2023-03-21 11:57:15,140 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:15,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41010 deadline: 1679399895136, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:15,140 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:15,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41068 deadline: 1679399895136, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:15,141 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:15,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41034 deadline: 1679399895137, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:15,144 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:15,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41056 deadline: 1679399895140, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:15,145 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:15,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41084 deadline: 1679399895140, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:15,245 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:15,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41068 deadline: 1679399895241, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:15,245 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:15,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41010 deadline: 1679399895241, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:15,246 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:15,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41034 deadline: 1679399895241, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:15,250 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:15,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41056 deadline: 1679399895245, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:15,250 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:15,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41084 deadline: 1679399895245, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:15,320 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=36, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/.tmp/A/fead8f3c4c9340758103019f767e8c95 2023-03-21 11:57:15,327 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=36 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/.tmp/B/44cbf0fe5a33402287b74db3f4be270c 2023-03-21 11:57:15,336 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=36 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/.tmp/C/721d4b9768f242c98a6af7e44f7339a0 2023-03-21 11:57:15,340 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/.tmp/A/fead8f3c4c9340758103019f767e8c95 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/A/fead8f3c4c9340758103019f767e8c95 2023-03-21 11:57:15,343 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/A/fead8f3c4c9340758103019f767e8c95, entries=150, sequenceid=36, filesize=25.1 K 2023-03-21 11:57:15,343 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/.tmp/B/44cbf0fe5a33402287b74db3f4be270c as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/B/44cbf0fe5a33402287b74db3f4be270c 2023-03-21 11:57:15,346 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/B/44cbf0fe5a33402287b74db3f4be270c, entries=150, sequenceid=36, filesize=11.6 K 2023-03-21 11:57:15,347 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/.tmp/C/721d4b9768f242c98a6af7e44f7339a0 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/C/721d4b9768f242c98a6af7e44f7339a0 2023-03-21 11:57:15,349 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/C/721d4b9768f242c98a6af7e44f7339a0, entries=150, sequenceid=36, filesize=11.6 K 2023-03-21 11:57:15,350 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for c7c660a43107606cd80c805b0450f72f in 464ms, sequenceid=36, compaction requested=false 2023-03-21 11:57:15,350 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c7c660a43107606cd80c805b0450f72f: 2023-03-21 11:57:15,350 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. 2023-03-21 11:57:15,350 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:57:15,350 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:57:15,350 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:15,350 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:57:15,350 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:57:15,351 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:15,351 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:15,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:15,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:15,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:15,351 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:15,352 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:57:15,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:15,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:15,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:15,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:15,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:15,352 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:57:15,352 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6854fcf9[Count = 0] remaining members to acquire global barrier 2023-03-21 11:57:15,352 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:57:15,353 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:15,353 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:15,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:15,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:15,353 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:57:15,353 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:15,353 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:57:15,353 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:57:15,353 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:57:15,355 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:57:15,355 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:15,355 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:15,355 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:57:15,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:15,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:15,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:15,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:15,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:15,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:15,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:15,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:15,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:15,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:15,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:57:15,359 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:57:15,359 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:57:15,359 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:57:15,359 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:57:15,359 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:57:15,359 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:57:15,359 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:15,359 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:15,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:15,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:15,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:15,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:15,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:15,361 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:15,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:15,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:15,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:15,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:15,361 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:15,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:15,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:15,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:15,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:15,363 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:15,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:15,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:15,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:15,364 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:15,364 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:15,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:15,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:15,365 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:15,365 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:15,365 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:15,365 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:57:15,365 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:57:15,365 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:15,365 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:15,365 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:15,365 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:15,365 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:15,365 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:57:15,365 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:57:15,365 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:57:15,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:15,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:15,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(9158): Flush requested on c7c660a43107606cd80c805b0450f72f 2023-03-21 11:57:15,450 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c7c660a43107606cd80c805b0450f72f 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-03-21 11:57:15,450 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c7c660a43107606cd80c805b0450f72f, store=A 2023-03-21 11:57:15,451 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:57:15,451 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c7c660a43107606cd80c805b0450f72f, store=B 2023-03-21 11:57:15,451 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:57:15,451 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c7c660a43107606cd80c805b0450f72f, store=C 2023-03-21 11:57:15,451 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:57:15,465 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:15,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:15,467 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:57:15,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:57:15,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:57:15,472 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:57:15,472 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:15,472 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:57:15,472 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:57:15,472 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:15,472 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:15,473 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:15,473 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:15,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:15,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:15,473 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:15,473 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:57:15,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:15,473 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:15,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:57:15,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:15,473 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:57:15,474 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:57:15,474 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:57:15,474 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:57:15,474 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:15,474 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:57:15,478 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:15,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41034 deadline: 1679399895467, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:15,482 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:57:15,482 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:57:15,482 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. 2023-03-21 11:57:15,482 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. started... 2023-03-21 11:57:15,482 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. as already flushing 2023-03-21 11:57:15,482 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. 2023-03-21 11:57:15,482 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:57:15,482 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:57:15,482 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:15,486 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:57:15,486 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:57:15,487 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:15,487 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:15,487 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:15,487 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:57:15,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:15,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:15,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:15,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:15,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:15,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:15,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:15,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:15,488 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:15,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41056 deadline: 1679399895469, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:15,489 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:57:15,489 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4404a51[Count = 0] remaining members to acquire global barrier 2023-03-21 11:57:15,489 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:15,489 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:57:15,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41084 deadline: 1679399895470, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:15,489 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:15,489 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:15,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41068 deadline: 1679399895478, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:15,489 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:15,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41010 deadline: 1679399895478, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:15,489 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:15,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:15,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:15,490 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:57:15,490 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:57:15,490 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:57:15,490 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:15,491 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:57:15,492 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:15,492 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:57:15,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:15,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:15,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:15,492 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:15,492 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:57:15,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:15,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:15,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:15,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:15,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:15,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:15,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:15,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:57:15,494 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:57:15,494 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:57:15,494 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:57:15,494 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:57:15,494 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:57:15,494 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:57:15,495 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:15,495 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:15,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:15,495 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:15,495 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:15,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:15,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:15,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:15,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:15,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:15,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:15,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:15,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:15,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:15,496 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:15,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:15,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:15,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:15,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:15,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:15,497 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:15,497 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:15,498 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:15,498 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:57:15,498 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:57:15,498 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:15,498 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:57:15,498 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:15,498 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:15,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:15,498 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:15,498 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:57:15,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:15,498 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:15,498 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:57:15,498 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:15,498 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:15,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:15,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:15,585 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:15,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41034 deadline: 1679399895579, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:15,594 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:15,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41084 deadline: 1679399895590, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:15,595 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:15,595 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:15,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41010 deadline: 1679399895590, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:15,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41068 deadline: 1679399895591, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:15,595 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:15,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41056 deadline: 1679399895591, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:15,598 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:15,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:15,600 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:57:15,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:57:15,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:57:15,601 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:57:15,601 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:15,601 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:57:15,601 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:57:15,601 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:15,601 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:15,602 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:15,602 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:15,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:15,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:15,602 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:15,602 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:57:15,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:15,602 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:15,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:57:15,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:15,603 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:57:15,603 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:57:15,603 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:57:15,603 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:57:15,603 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:15,603 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:57:15,603 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:57:15,603 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:57:15,603 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. 2023-03-21 11:57:15,603 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. started... 2023-03-21 11:57:15,603 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. as already flushing 2023-03-21 11:57:15,603 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. 2023-03-21 11:57:15,603 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:57:15,603 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:57:15,603 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:15,603 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:57:15,603 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:57:15,604 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:15,604 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:15,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:15,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:15,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:15,604 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:15,604 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:57:15,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:15,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:15,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:15,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:15,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:15,606 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:57:15,606 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ebe1caf[Count = 0] remaining members to acquire global barrier 2023-03-21 11:57:15,606 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:57:15,606 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:15,606 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:15,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:15,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:15,607 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:57:15,607 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:57:15,607 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:57:15,607 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:15,607 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:57:15,607 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:15,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:15,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:15,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:15,608 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:57:15,608 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:15,608 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:57:15,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:15,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:15,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:15,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:15,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:15,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:15,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:15,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:57:15,609 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:57:15,609 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:57:15,609 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:57:15,609 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:57:15,609 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:57:15,609 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:57:15,610 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:15,610 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:15,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:15,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:15,610 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:15,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:15,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:15,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:15,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:15,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:15,610 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:15,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:15,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:15,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:15,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:15,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:15,611 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:15,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:15,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:15,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:15,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:15,612 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:15,613 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:15,613 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:15,613 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:15,613 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:15,613 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:57:15,613 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:57:15,613 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:15,613 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:15,613 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:15,613 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:57:15,613 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:15,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:15,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:15,613 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:57:15,613 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:57:15,613 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:15,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:15,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:15,618 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-03-21 11:57:15,713 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:15,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:15,715 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:57:15,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:57:15,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:57:15,716 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:57:15,716 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:15,716 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:57:15,716 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:57:15,716 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:15,716 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:15,717 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:15,717 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:15,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:15,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:15,717 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:15,717 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:57:15,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:15,718 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:15,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:57:15,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:15,718 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:57:15,718 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:57:15,718 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:57:15,718 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:57:15,718 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:15,718 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:57:15,718 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:57:15,718 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:57:15,718 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. 2023-03-21 11:57:15,718 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. started... 2023-03-21 11:57:15,719 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. as already flushing 2023-03-21 11:57:15,719 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. 2023-03-21 11:57:15,719 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:57:15,719 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:57:15,719 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:15,719 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:57:15,719 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:57:15,720 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:15,720 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:15,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:15,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:15,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:15,720 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:15,720 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:57:15,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:15,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:15,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:15,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:15,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:15,721 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:57:15,721 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:57:15,721 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5fcfafce[Count = 0] remaining members to acquire global barrier 2023-03-21 11:57:15,721 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:15,721 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:15,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:15,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:15,721 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:57:15,721 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:15,722 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:57:15,721 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:57:15,722 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:57:15,722 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:57:15,722 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:15,722 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:15,722 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:57:15,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:15,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:15,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:15,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:15,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:15,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:15,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:15,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:15,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:15,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:15,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:57:15,724 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:57:15,724 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:57:15,724 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:57:15,724 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:57:15,724 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:57:15,724 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:57:15,725 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:15,725 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:15,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:15,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:15,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:15,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:15,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:15,725 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:15,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:15,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:15,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:15,725 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:15,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:15,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:15,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:15,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:15,727 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:15,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:15,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:15,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:15,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:15,728 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:15,728 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:15,728 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:15,728 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:15,728 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:15,728 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:15,728 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:15,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:15,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:15,728 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:15,728 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:57:15,728 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:57:15,728 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:15,729 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:15,729 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:57:15,729 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:57:15,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:15,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:15,729 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:57:15,788 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:15,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41034 deadline: 1679399895786, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:15,798 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:15,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41010 deadline: 1679399895795, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:15,798 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:15,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41068 deadline: 1679399895795, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:15,798 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:15,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41084 deadline: 1679399895796, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:15,798 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:15,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41056 deadline: 1679399895796, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:15,829 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:15,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:15,831 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:57:15,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:57:15,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:57:15,831 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:57:15,832 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:15,832 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:57:15,832 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:57:15,832 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:15,832 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:15,833 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:15,833 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:15,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:15,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:15,833 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:15,833 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:57:15,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:15,833 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:15,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:57:15,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:15,833 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:57:15,833 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:57:15,834 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:57:15,834 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:57:15,834 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:15,834 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:57:15,834 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:57:15,834 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:57:15,834 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. 2023-03-21 11:57:15,834 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. started... 2023-03-21 11:57:15,834 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. as already flushing 2023-03-21 11:57:15,834 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. 2023-03-21 11:57:15,834 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:57:15,834 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:57:15,834 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:15,834 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:57:15,834 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:57:15,835 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:15,835 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:15,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:15,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:15,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:15,835 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:15,835 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:57:15,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:15,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:15,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:15,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:15,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:15,836 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:57:15,836 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@10b323fa[Count = 0] remaining members to acquire global barrier 2023-03-21 11:57:15,836 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:57:15,836 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:15,837 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:15,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:15,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:15,837 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:57:15,837 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:57:15,837 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:57:15,837 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:15,837 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:57:15,837 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:15,837 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:57:15,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:15,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:15,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:15,838 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:15,838 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:57:15,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:15,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:15,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:15,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:15,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:15,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:15,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:15,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:57:15,841 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:57:15,841 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:57:15,841 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:57:15,841 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:57:15,841 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:57:15,841 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:57:15,841 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:15,842 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:15,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:15,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:15,842 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:15,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:15,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:15,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:15,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:15,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:15,842 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:15,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:15,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:15,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:15,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:15,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:15,843 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:15,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:15,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:15,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:15,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:15,845 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:15,845 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:15,845 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:15,845 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:15,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:15,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:15,845 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:15,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:15,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:15,846 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:15,846 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:15,846 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:15,846 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:15,846 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:15,846 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:57:15,846 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:57:15,846 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:57:15,846 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:57:15,846 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:57:15,890 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303210d3f9931a8e144fba951e07d7f73edf0 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303210d3f9931a8e144fba951e07d7f73edf0 2023-03-21 11:57:15,895 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=53, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/.tmp/A/642fec24c0934a678f032544cbf9db7f 2023-03-21 11:57:15,946 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:15,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:15,948 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:57:15,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:57:15,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:57:15,949 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:57:15,949 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:15,949 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:57:15,949 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:57:15,949 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:15,949 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:15,950 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:15,950 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:15,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:15,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:15,950 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:15,950 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:57:15,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:15,950 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:15,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:57:15,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:15,950 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:57:15,951 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:57:15,951 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:57:15,951 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:57:15,951 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:15,951 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:57:15,951 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:57:15,951 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:57:15,951 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. 2023-03-21 11:57:15,951 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. started... 2023-03-21 11:57:15,951 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. as already flushing 2023-03-21 11:57:15,951 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. 2023-03-21 11:57:15,951 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:57:15,951 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:57:15,951 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:15,951 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:57:15,951 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:57:15,952 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:15,952 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:15,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:15,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:15,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:15,952 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:15,952 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:57:15,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:15,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:15,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:15,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:15,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:15,954 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:57:15,954 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:57:15,954 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16a7e17f[Count = 0] remaining members to acquire global barrier 2023-03-21 11:57:15,954 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:15,954 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:15,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:15,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:15,954 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:57:15,954 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:57:15,954 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:57:15,954 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:15,954 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:57:15,955 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:57:15,955 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:15,955 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:15,955 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:57:15,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:15,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:15,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:15,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:15,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:15,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:15,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:15,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:15,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:15,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:15,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:57:15,957 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:57:15,957 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:57:15,957 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:57:15,957 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:57:15,957 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:57:15,957 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:57:15,958 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:15,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:15,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:15,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:15,958 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:15,958 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:15,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:15,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:15,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:15,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:15,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:15,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:15,959 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:15,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:15,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:15,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:15,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:15,960 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:15,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:15,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:15,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:15,961 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:15,961 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:15,961 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:57:15,961 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:57:15,961 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:15,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:15,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:15,961 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:57:15,962 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:57:15,962 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:57:15,962 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:15,962 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:15,962 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:15,962 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:15,962 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:15,962 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:15,962 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:15,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:15,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:16,062 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:16,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:16,064 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:57:16,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:57:16,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:57:16,064 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:57:16,064 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:16,065 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:57:16,065 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:57:16,065 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:16,065 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:16,065 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:16,066 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:16,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:16,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:16,066 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:16,066 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:57:16,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:16,066 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:16,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:57:16,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:16,066 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:57:16,067 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:57:16,067 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:57:16,067 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:57:16,067 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:16,067 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:57:16,067 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:57:16,067 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:57:16,067 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. 2023-03-21 11:57:16,067 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. started... 2023-03-21 11:57:16,067 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. as already flushing 2023-03-21 11:57:16,067 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. 2023-03-21 11:57:16,067 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:57:16,067 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:57:16,067 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:16,067 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:57:16,067 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:57:16,068 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:16,068 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:16,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:16,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:16,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:16,068 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:16,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:16,068 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:57:16,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:16,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:16,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:16,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:16,069 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:57:16,069 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:57:16,069 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@8e2cc60[Count = 0] remaining members to acquire global barrier 2023-03-21 11:57:16,069 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:16,070 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:16,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:16,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:16,070 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:16,070 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:57:16,070 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:57:16,070 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:57:16,070 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:57:16,071 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:16,071 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:57:16,071 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:16,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:16,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:16,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:16,071 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:57:16,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:16,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:16,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:16,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:16,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:16,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:16,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:16,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:57:16,073 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:57:16,073 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:57:16,073 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:57:16,073 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:57:16,073 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:57:16,073 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:57:16,074 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:16,074 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:16,074 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:16,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:16,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:16,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:16,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:16,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:16,074 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:16,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:16,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:16,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:16,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:16,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:16,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:16,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:16,075 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:16,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:16,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:16,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:16,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:16,076 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:16,076 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:16,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:16,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:16,076 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:16,076 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:16,076 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:16,076 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:16,076 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:16,076 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:16,076 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:16,076 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:57:16,076 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:57:16,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:16,076 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:16,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:16,077 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:57:16,077 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:57:16,077 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:57:16,092 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:16,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41034 deadline: 1679399896089, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:16,102 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:16,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41010 deadline: 1679399896099, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:16,103 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:16,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41084 deadline: 1679399896100, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:16,103 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:16,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41056 deadline: 1679399896100, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:16,103 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:16,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41068 deadline: 1679399896100, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:16,177 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:16,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:16,179 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:57:16,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:57:16,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:57:16,180 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:57:16,180 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:16,180 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:57:16,180 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:57:16,180 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:16,180 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:16,181 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:16,181 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:16,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:16,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:16,181 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:16,181 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:57:16,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:16,181 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:16,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:57:16,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:16,182 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:57:16,182 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:57:16,182 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:57:16,182 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:57:16,182 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:16,182 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:57:16,182 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:57:16,182 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:57:16,182 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. 2023-03-21 11:57:16,182 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. started... 2023-03-21 11:57:16,182 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. as already flushing 2023-03-21 11:57:16,182 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. 2023-03-21 11:57:16,182 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:57:16,182 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:57:16,182 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:16,182 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:57:16,182 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:57:16,183 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:16,183 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:16,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:16,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:16,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:16,183 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:16,183 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:57:16,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:16,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:16,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:16,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:16,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:16,186 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:57:16,186 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:57:16,186 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2a71b36d[Count = 0] remaining members to acquire global barrier 2023-03-21 11:57:16,186 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:16,186 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:16,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:16,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:16,187 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:57:16,187 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:57:16,187 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:57:16,187 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:16,187 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:57:16,187 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:16,187 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:57:16,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:16,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:16,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:16,187 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:16,187 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:57:16,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:16,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:16,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:16,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:16,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:16,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:16,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:16,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:57:16,189 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:57:16,189 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:57:16,189 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:57:16,189 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:57:16,189 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:57:16,189 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:57:16,190 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:16,190 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:16,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:16,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:16,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:16,190 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:16,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:16,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:16,190 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:16,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:16,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:16,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:16,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:16,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:16,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:16,191 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:16,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:16,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:16,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:16,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:16,193 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:16,193 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:16,193 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:16,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:16,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:16,193 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:16,193 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:16,193 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:57:16,194 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:57:16,193 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:16,194 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:16,194 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:16,194 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:16,194 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:57:16,194 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:16,194 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:57:16,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:16,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:16,194 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:57:16,294 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:16,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:16,296 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:57:16,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:57:16,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:57:16,298 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:57:16,298 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:16,298 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:57:16,298 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:57:16,298 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:16,298 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:16,299 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:16,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:16,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:16,299 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:16,300 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:16,300 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:57:16,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:16,300 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:16,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:57:16,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:16,300 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:57:16,300 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:57:16,300 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:57:16,300 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:57:16,300 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:16,300 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:57:16,301 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:57:16,301 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:57:16,301 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. 2023-03-21 11:57:16,301 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. started... 2023-03-21 11:57:16,301 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. as already flushing 2023-03-21 11:57:16,301 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. 2023-03-21 11:57:16,301 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:57:16,301 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:57:16,301 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:16,301 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:57:16,301 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:57:16,302 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:16,302 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:16,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:16,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:16,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:16,302 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:16,302 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:57:16,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:16,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:16,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:16,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:16,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:16,303 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:57:16,303 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@23d4f92a[Count = 0] remaining members to acquire global barrier 2023-03-21 11:57:16,303 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:57:16,303 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:16,303 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=53 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/.tmp/B/34732a5e76eb402fb867f57d9848d0cc 2023-03-21 11:57:16,304 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:16,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:16,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:16,304 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:57:16,304 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:57:16,304 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:57:16,304 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:16,304 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:57:16,305 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:16,305 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:57:16,305 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:16,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:16,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:16,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:16,305 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:57:16,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:16,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:16,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:16,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:16,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:16,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:16,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:16,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:57:16,307 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:57:16,307 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:57:16,307 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:57:16,307 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:57:16,307 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:57:16,308 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:57:16,308 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:16,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:16,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:16,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:16,309 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:16,309 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:16,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:16,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:16,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:16,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:16,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:16,309 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:16,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:16,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:16,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:16,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:16,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:16,310 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:16,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:16,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:16,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:16,312 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:16,312 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:16,312 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:16,312 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:16,312 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:16,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:16,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:16,312 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:16,312 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:57:16,312 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:57:16,312 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:57:16,312 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:16,312 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:16,312 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:57:16,312 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:16,312 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:57:16,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:16,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:16,312 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:16,313 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=53 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/.tmp/C/ae4e6ef53e28464ba4e7ab28b17517f1 2023-03-21 11:57:16,316 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/.tmp/A/642fec24c0934a678f032544cbf9db7f as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/A/642fec24c0934a678f032544cbf9db7f 2023-03-21 11:57:16,319 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/A/642fec24c0934a678f032544cbf9db7f, entries=300, sequenceid=53, filesize=45.5 K 2023-03-21 11:57:16,320 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/.tmp/B/34732a5e76eb402fb867f57d9848d0cc as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/B/34732a5e76eb402fb867f57d9848d0cc 2023-03-21 11:57:16,327 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/B/34732a5e76eb402fb867f57d9848d0cc, entries=150, sequenceid=53, filesize=11.6 K 2023-03-21 11:57:16,327 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/.tmp/C/ae4e6ef53e28464ba4e7ab28b17517f1 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/C/ae4e6ef53e28464ba4e7ab28b17517f1 2023-03-21 11:57:16,331 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/C/ae4e6ef53e28464ba4e7ab28b17517f1, entries=150, sequenceid=53, filesize=11.6 K 2023-03-21 11:57:16,332 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for c7c660a43107606cd80c805b0450f72f in 882ms, sequenceid=53, compaction requested=true 2023-03-21 11:57:16,332 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c7c660a43107606cd80c805b0450f72f: 2023-03-21 11:57:16,332 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-21 11:57:16,332 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-21 11:57:16,332 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-21 11:57:16,333 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-21 11:57:16,333 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-21 11:57:16,333 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-21 11:57:16,333 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 98062 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-21 11:57:16,333 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1912): c7c660a43107606cd80c805b0450f72f/B is initiating minor compaction (all files) 2023-03-21 11:57:16,333 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1912): c7c660a43107606cd80c805b0450f72f/A is initiating minor compaction (all files) 2023-03-21 11:57:16,333 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c7c660a43107606cd80c805b0450f72f/B in TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. 2023-03-21 11:57:16,333 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c7c660a43107606cd80c805b0450f72f/A in TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. 2023-03-21 11:57:16,334 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/B/6d7e3a7a2c0f4a19864a119375c9124b, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/B/44cbf0fe5a33402287b74db3f4be270c, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/B/34732a5e76eb402fb867f57d9848d0cc] into tmpdir=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/.tmp, totalSize=34.8 K 2023-03-21 11:57:16,334 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/A/70b70bb7cbf84daabcbc34626c25b38f, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/A/fead8f3c4c9340758103019f767e8c95, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/A/642fec24c0934a678f032544cbf9db7f] into tmpdir=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/.tmp, totalSize=95.8 K 2023-03-21 11:57:16,334 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 70b70bb7cbf84daabcbc34626c25b38f, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=18, earliestPutTs=1679399833983 2023-03-21 11:57:16,334 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 6d7e3a7a2c0f4a19864a119375c9124b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=18, earliestPutTs=1679399833983 2023-03-21 11:57:16,334 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting fead8f3c4c9340758103019f767e8c95, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=36, earliestPutTs=1679399834000 2023-03-21 11:57:16,338 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 44cbf0fe5a33402287b74db3f4be270c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=36, earliestPutTs=1679399834000 2023-03-21 11:57:16,339 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 642fec24c0934a678f032544cbf9db7f, keycount=300, bloomtype=ROW, size=45.5 K, encoding=NONE, compression=NONE, seqNum=53, earliestPutTs=1679399835136 2023-03-21 11:57:16,339 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 34732a5e76eb402fb867f57d9848d0cc, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=53, earliestPutTs=1679399835136 2023-03-21 11:57:16,350 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] throttle.PressureAwareThroughputController(145): c7c660a43107606cd80c805b0450f72f#B#compaction#292 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-21 11:57:16,352 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] throttle.PressureAwareThroughputController(145): c7c660a43107606cd80c805b0450f72f#A#compaction#291 average throughput is 6.57 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-21 11:57:16,364 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/.tmp/B/74bb072119614bc8826634d4ef34469c as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/B/74bb072119614bc8826634d4ef34469c 2023-03-21 11:57:16,368 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c7c660a43107606cd80c805b0450f72f/B of c7c660a43107606cd80c805b0450f72f into 74bb072119614bc8826634d4ef34469c(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-21 11:57:16,368 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c7c660a43107606cd80c805b0450f72f: 2023-03-21 11:57:16,368 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f., storeName=c7c660a43107606cd80c805b0450f72f/B, priority=13, startTime=1679399836332; duration=0sec 2023-03-21 11:57:16,368 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-21 11:57:16,369 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-21 11:57:16,369 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-21 11:57:16,369 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1912): c7c660a43107606cd80c805b0450f72f/C is initiating minor compaction (all files) 2023-03-21 11:57:16,369 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c7c660a43107606cd80c805b0450f72f/C in TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. 2023-03-21 11:57:16,369 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/C/c7e173a2da7346f79762e26607b8698b, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/C/721d4b9768f242c98a6af7e44f7339a0, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/C/ae4e6ef53e28464ba4e7ab28b17517f1] into tmpdir=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/.tmp, totalSize=34.8 K 2023-03-21 11:57:16,370 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting c7e173a2da7346f79762e26607b8698b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=18, earliestPutTs=1679399833983 2023-03-21 11:57:16,370 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 721d4b9768f242c98a6af7e44f7339a0, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=36, earliestPutTs=1679399834000 2023-03-21 11:57:16,370 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting ae4e6ef53e28464ba4e7ab28b17517f1, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=53, earliestPutTs=1679399835136 2023-03-21 11:57:16,374 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] throttle.PressureAwareThroughputController(145): c7c660a43107606cd80c805b0450f72f#C#compaction#293 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-21 11:57:16,412 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:16,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:16,414 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:57:16,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:57:16,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:57:16,415 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:57:16,415 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:16,415 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:57:16,415 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:57:16,415 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:16,415 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:16,416 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:16,416 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:16,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:16,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:16,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:16,416 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:16,416 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:57:16,417 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:16,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:57:16,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:16,417 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:57:16,417 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:57:16,417 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:57:16,417 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:57:16,417 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:16,417 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:57:16,417 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:57:16,417 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:57:16,417 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. 2023-03-21 11:57:16,417 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. started... 2023-03-21 11:57:16,417 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c7c660a43107606cd80c805b0450f72f 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-03-21 11:57:16,418 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c7c660a43107606cd80c805b0450f72f, store=A 2023-03-21 11:57:16,418 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:57:16,418 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c7c660a43107606cd80c805b0450f72f, store=B 2023-03-21 11:57:16,418 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:57:16,418 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c7c660a43107606cd80c805b0450f72f, store=C 2023-03-21 11:57:16,418 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:57:16,432 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230321517e7a1441ce46e6b598c15c88621bd3 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230321517e7a1441ce46e6b598c15c88621bd3 2023-03-21 11:57:16,440 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=69, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/.tmp/A/eac2a454f4f8402482e4c0b851064b97 2023-03-21 11:57:16,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(9158): Flush requested on c7c660a43107606cd80c805b0450f72f 2023-03-21 11:57:16,596 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. as already flushing 2023-03-21 11:57:16,622 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:16,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41056 deadline: 1679399896615, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:16,622 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:16,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41068 deadline: 1679399896616, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:16,627 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:16,627 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:16,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41084 deadline: 1679399896621, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:16,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41034 deadline: 1679399896617, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:16,627 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:16,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41010 deadline: 1679399896622, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:16,725 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:16,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41068 deadline: 1679399896723, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:16,725 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:16,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41056 deadline: 1679399896723, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:16,730 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:16,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41084 deadline: 1679399896727, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:16,731 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:16,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41034 deadline: 1679399896728, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:16,731 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:16,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41010 deadline: 1679399896728, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:16,781 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/.tmp/C/ad148fb307754ab2a9c23757fdfec04a as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/C/ad148fb307754ab2a9c23757fdfec04a 2023-03-21 11:57:16,785 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c7c660a43107606cd80c805b0450f72f/C of c7c660a43107606cd80c805b0450f72f into ad148fb307754ab2a9c23757fdfec04a(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-21 11:57:16,785 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c7c660a43107606cd80c805b0450f72f: 2023-03-21 11:57:16,785 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f., storeName=c7c660a43107606cd80c805b0450f72f/C, priority=13, startTime=1679399836333; duration=0sec 2023-03-21 11:57:16,785 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-21 11:57:16,854 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=69 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/.tmp/B/991eadee7bb34891bc0965cc68573ab9 2023-03-21 11:57:16,929 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:16,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41056 deadline: 1679399896926, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:16,929 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:16,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41068 deadline: 1679399896927, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:16,935 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:16,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41084 deadline: 1679399896932, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:16,935 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:16,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41034 deadline: 1679399896932, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:16,935 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:16,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41010 deadline: 1679399896932, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:17,234 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:17,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41068 deadline: 1679399897230, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:17,236 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:17,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41056 deadline: 1679399897232, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:17,241 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:17,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41010 deadline: 1679399897236, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:17,241 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:17,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41034 deadline: 1679399897236, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:17,243 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:17,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41084 deadline: 1679399897238, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:17,283 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=69 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/.tmp/C/b16d3d1073cb49b1b291c223cec3d525 2023-03-21 11:57:17,286 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/.tmp/A/eac2a454f4f8402482e4c0b851064b97 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/A/eac2a454f4f8402482e4c0b851064b97 2023-03-21 11:57:17,290 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/A/eac2a454f4f8402482e4c0b851064b97, entries=150, sequenceid=69, filesize=25.1 K 2023-03-21 11:57:17,291 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/.tmp/B/991eadee7bb34891bc0965cc68573ab9 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/B/991eadee7bb34891bc0965cc68573ab9 2023-03-21 11:57:17,294 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/B/991eadee7bb34891bc0965cc68573ab9, entries=150, sequenceid=69, filesize=11.6 K 2023-03-21 11:57:17,296 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/.tmp/C/b16d3d1073cb49b1b291c223cec3d525 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/C/b16d3d1073cb49b1b291c223cec3d525 2023-03-21 11:57:17,300 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/C/b16d3d1073cb49b1b291c223cec3d525, entries=150, sequenceid=69, filesize=11.6 K 2023-03-21 11:57:17,300 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for c7c660a43107606cd80c805b0450f72f in 883ms, sequenceid=69, compaction requested=false 2023-03-21 11:57:17,303 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c7c660a43107606cd80c805b0450f72f: 2023-03-21 11:57:17,303 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. 2023-03-21 11:57:17,303 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:57:17,303 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:57:17,303 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:17,303 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:57:17,303 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:57:17,326 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:17,326 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:17,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:17,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:17,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:17,327 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:17,327 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:57:17,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:17,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:17,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:17,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:17,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:17,327 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:57:17,328 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ad15836[Count = 0] remaining members to acquire global barrier 2023-03-21 11:57:17,328 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:57:17,328 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:17,328 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:17,328 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:57:17,329 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:17,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:17,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:17,329 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:57:17,329 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:57:17,329 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:57:17,329 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:17,329 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:57:17,330 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:17,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:17,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:17,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:17,330 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:57:17,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:17,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:17,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:17,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:17,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:17,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:17,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:17,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:57:17,331 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:57:17,331 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:57:17,331 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:57:17,331 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:57:17,331 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:57:17,331 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:57:17,332 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:17,332 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:17,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:17,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:17,332 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:17,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:17,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:17,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:17,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:17,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:17,333 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:17,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:17,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:17,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:17,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:17,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:17,333 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:17,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:17,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:17,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:17,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:17,335 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:17,335 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:17,335 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:17,335 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:17,335 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:57:17,335 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:17,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:17,335 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:57:17,335 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:57:17,335 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:17,335 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:17,335 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:17,335 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:17,335 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:17,335 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:17,335 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:17,335 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:57:17,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:17,335 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:57:17,435 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:17,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:17,437 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:57:17,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:57:17,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:57:17,438 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:57:17,438 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:17,438 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:57:17,438 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:57:17,438 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:17,438 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:17,439 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:17,439 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:17,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:17,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:17,439 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:17,439 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:57:17,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:17,439 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:17,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:57:17,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:17,440 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:57:17,440 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:57:17,440 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:57:17,440 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:57:17,440 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:17,440 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:57:17,440 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:57:17,440 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:57:17,440 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. 2023-03-21 11:57:17,440 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. started... 2023-03-21 11:57:17,440 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c7c660a43107606cd80c805b0450f72f 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-03-21 11:57:17,441 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c7c660a43107606cd80c805b0450f72f, store=A 2023-03-21 11:57:17,441 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:57:17,441 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c7c660a43107606cd80c805b0450f72f, store=B 2023-03-21 11:57:17,441 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:57:17,441 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c7c660a43107606cd80c805b0450f72f, store=C 2023-03-21 11:57:17,441 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:57:17,580 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/.tmp/A/762370c7f513458aa58cc249ca0eabc1 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/A/762370c7f513458aa58cc249ca0eabc1 2023-03-21 11:57:17,583 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c7c660a43107606cd80c805b0450f72f/A of c7c660a43107606cd80c805b0450f72f into 762370c7f513458aa58cc249ca0eabc1(size=25.2 K), total size for store is 50.4 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-03-21 11:57:17,583 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c7c660a43107606cd80c805b0450f72f: 2023-03-21 11:57:17,583 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f., storeName=c7c660a43107606cd80c805b0450f72f/A, priority=13, startTime=1679399836332; duration=1sec 2023-03-21 11:57:17,583 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-21 11:57:17,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(9158): Flush requested on c7c660a43107606cd80c805b0450f72f 2023-03-21 11:57:17,737 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. as already flushing 2023-03-21 11:57:17,756 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:17,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41084 deadline: 1679399897750, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:17,756 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:17,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41068 deadline: 1679399897750, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:17,756 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:17,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41010 deadline: 1679399897751, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:17,757 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:17,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41056 deadline: 1679399897752, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:17,757 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:17,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41034 deadline: 1679399897753, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:17,853 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230321ed03bea8e6b84960a675bb9b505937f2 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230321ed03bea8e6b84960a675bb9b505937f2 2023-03-21 11:57:17,859 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=87, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/.tmp/A/e260dd6de6164fdaa731e17ff288ce0e 2023-03-21 11:57:17,862 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:17,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41068 deadline: 1679399897859, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:17,863 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:17,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41084 deadline: 1679399897859, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:17,863 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:17,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41034 deadline: 1679399897859, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:17,863 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:17,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41056 deadline: 1679399897859, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:17,863 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:17,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41010 deadline: 1679399897859, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:18,068 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:18,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41034 deadline: 1679399898064, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:18,068 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:18,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41010 deadline: 1679399898064, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:18,068 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:18,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41068 deadline: 1679399898064, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:18,068 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:18,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41084 deadline: 1679399898064, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:18,068 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:18,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41056 deadline: 1679399898065, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:18,269 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=87 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/.tmp/B/1e6754799ea84d7b9bf9c471e3b74350 2023-03-21 11:57:18,372 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:18,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41034 deadline: 1679399898369, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:18,372 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:18,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41010 deadline: 1679399898369, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:18,374 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:18,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41068 deadline: 1679399898371, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:18,375 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:18,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41084 deadline: 1679399898371, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:18,375 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:18,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41056 deadline: 1679399898371, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:18,677 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=87 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/.tmp/C/37e03c26e78c41379b2738a51d19d6ba 2023-03-21 11:57:18,681 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/.tmp/A/e260dd6de6164fdaa731e17ff288ce0e as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/A/e260dd6de6164fdaa731e17ff288ce0e 2023-03-21 11:57:18,683 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/A/e260dd6de6164fdaa731e17ff288ce0e, entries=150, sequenceid=87, filesize=25.1 K 2023-03-21 11:57:18,684 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/.tmp/B/1e6754799ea84d7b9bf9c471e3b74350 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/B/1e6754799ea84d7b9bf9c471e3b74350 2023-03-21 11:57:18,694 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/B/1e6754799ea84d7b9bf9c471e3b74350, entries=150, sequenceid=87, filesize=11.6 K 2023-03-21 11:57:18,698 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/.tmp/C/37e03c26e78c41379b2738a51d19d6ba as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/C/37e03c26e78c41379b2738a51d19d6ba 2023-03-21 11:57:18,701 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/C/37e03c26e78c41379b2738a51d19d6ba, entries=150, sequenceid=87, filesize=11.6 K 2023-03-21 11:57:18,702 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for c7c660a43107606cd80c805b0450f72f in 1262ms, sequenceid=87, compaction requested=true 2023-03-21 11:57:18,702 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c7c660a43107606cd80c805b0450f72f: 2023-03-21 11:57:18,702 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. 2023-03-21 11:57:18,705 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:57:18,705 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:57:18,705 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:18,705 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:57:18,706 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:57:18,706 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:18,706 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:18,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:18,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:18,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:18,707 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:18,707 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:57:18,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:18,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:18,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:18,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:18,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:18,708 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:57:18,708 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@14ce14c0[Count = 0] remaining members to acquire global barrier 2023-03-21 11:57:18,708 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:57:18,708 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:18,708 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:18,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:18,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:18,708 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:57:18,708 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:57:18,708 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:57:18,708 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:18,708 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:57:18,709 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:18,709 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:57:18,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:18,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:18,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:18,709 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:18,709 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:57:18,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:18,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:18,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:18,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:18,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:18,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:18,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:18,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:57:18,712 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:57:18,712 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:57:18,712 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:57:18,712 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:57:18,712 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:57:18,712 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:57:18,713 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:18,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:18,713 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:18,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:18,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:18,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:18,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:18,713 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:18,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:18,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:18,714 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:18,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:18,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:18,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:18,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:18,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:18,715 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:18,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:18,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:18,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:18,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:18,716 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:18,716 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:18,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:18,717 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:18,716 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:18,717 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:57:18,717 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:57:18,717 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:57:18,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:18,716 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:18,717 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:57:18,717 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:57:18,717 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:18,717 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:18,717 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:18,717 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:18,717 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:18,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:18,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:18,817 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:18,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:18,819 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:57:18,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:57:18,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:57:18,819 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:57:18,820 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:18,820 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:57:18,820 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:57:18,820 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:18,820 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:18,821 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:18,821 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:18,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:18,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:18,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:18,821 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:18,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:57:18,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:18,821 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:57:18,821 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:57:18,822 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:57:18,822 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:57:18,822 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:18,822 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:57:18,822 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:57:18,822 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:57:18,822 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. 2023-03-21 11:57:18,822 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. started... 2023-03-21 11:57:18,822 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c7c660a43107606cd80c805b0450f72f 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-03-21 11:57:18,822 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c7c660a43107606cd80c805b0450f72f, store=A 2023-03-21 11:57:18,822 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:57:18,822 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c7c660a43107606cd80c805b0450f72f, store=B 2023-03-21 11:57:18,822 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:57:18,822 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c7c660a43107606cd80c805b0450f72f, store=C 2023-03-21 11:57:18,823 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:57:18,822 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:18,825 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:57:18,835 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032197af70b8569848d99a72b9b5cfedd0bc to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032197af70b8569848d99a72b9b5cfedd0bc 2023-03-21 11:57:18,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(9158): Flush requested on c7c660a43107606cd80c805b0450f72f 2023-03-21 11:57:18,877 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. as already flushing 2023-03-21 11:57:18,897 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:18,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41034 deadline: 1679399898891, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:18,898 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:18,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41084 deadline: 1679399898892, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:18,898 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:18,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41010 deadline: 1679399898894, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:18,898 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:18,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41068 deadline: 1679399898895, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:18,903 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:18,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41056 deadline: 1679399898897, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:19,002 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:19,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41034 deadline: 1679399898998, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:19,002 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:19,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41084 deadline: 1679399898998, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:19,002 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:19,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41010 deadline: 1679399898998, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:19,003 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:19,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41068 deadline: 1679399898999, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:19,008 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:19,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41056 deadline: 1679399899003, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:19,206 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:19,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41034 deadline: 1679399899203, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:19,207 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:19,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41010 deadline: 1679399899203, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:19,207 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:19,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41084 deadline: 1679399899204, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:19,207 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:19,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41068 deadline: 1679399899204, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:19,216 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:19,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41056 deadline: 1679399899210, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:19,243 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=103, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/.tmp/A/b7247d124275490e90d3d0f3d4e5cb8b 2023-03-21 11:57:19,510 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:19,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41010 deadline: 1679399899508, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:19,514 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:19,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41034 deadline: 1679399899509, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:19,514 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:19,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41084 deadline: 1679399899510, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:19,514 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:19,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41068 deadline: 1679399899510, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:19,521 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:19,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41056 deadline: 1679399899518, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:19,650 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=103 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/.tmp/B/1a47576064c94558bba036d1819768a0 2023-03-21 11:57:19,659 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=103 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/.tmp/C/61d1a704c7584f7ebbfeb19cdd356840 2023-03-21 11:57:19,664 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/.tmp/A/b7247d124275490e90d3d0f3d4e5cb8b as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/A/b7247d124275490e90d3d0f3d4e5cb8b 2023-03-21 11:57:19,673 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/A/b7247d124275490e90d3d0f3d4e5cb8b, entries=150, sequenceid=103, filesize=25.1 K 2023-03-21 11:57:19,673 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/.tmp/B/1a47576064c94558bba036d1819768a0 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/B/1a47576064c94558bba036d1819768a0 2023-03-21 11:57:19,679 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/B/1a47576064c94558bba036d1819768a0, entries=150, sequenceid=103, filesize=11.6 K 2023-03-21 11:57:19,680 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/.tmp/C/61d1a704c7584f7ebbfeb19cdd356840 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/C/61d1a704c7584f7ebbfeb19cdd356840 2023-03-21 11:57:19,683 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/C/61d1a704c7584f7ebbfeb19cdd356840, entries=150, sequenceid=103, filesize=11.6 K 2023-03-21 11:57:19,683 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for c7c660a43107606cd80c805b0450f72f in 861ms, sequenceid=103, compaction requested=true 2023-03-21 11:57:19,683 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c7c660a43107606cd80c805b0450f72f: 2023-03-21 11:57:19,683 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. 2023-03-21 11:57:19,684 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:57:19,684 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:57:19,684 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:19,684 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:57:19,684 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:57:19,684 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:19,685 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:19,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:19,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:19,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:19,685 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:19,685 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:57:19,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:19,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:19,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:19,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:19,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:19,686 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:57:19,686 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@330337ef[Count = 0] remaining members to acquire global barrier 2023-03-21 11:57:19,686 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:57:19,686 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:19,686 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:19,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:19,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:19,686 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:57:19,687 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:57:19,687 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:57:19,687 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:19,687 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:57:19,687 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:57:19,687 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:19,687 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:19,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:19,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:19,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:19,687 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:57:19,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:19,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:19,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:19,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:19,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:19,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:19,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:19,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:57:19,689 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:57:19,689 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:57:19,689 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:57:19,689 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:57:19,689 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:57:19,689 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:57:19,690 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:19,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:19,690 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:19,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:19,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:19,690 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:19,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:19,690 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:19,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:19,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:19,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:19,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:19,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:19,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:19,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:19,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:19,692 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:19,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:19,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:19,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:19,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:19,693 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:19,693 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:19,693 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:19,693 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:19,693 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:19,693 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:19,693 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:57:19,693 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:19,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:19,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:19,693 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:57:19,694 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:57:19,693 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:19,694 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:19,694 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:19,694 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:57:19,694 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:57:19,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:19,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:19,794 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:19,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:19,796 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:57:19,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:57:19,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:57:19,797 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:57:19,797 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:19,797 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:57:19,797 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:57:19,797 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:19,797 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:19,799 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:19,799 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:19,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:19,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:19,799 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:19,799 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:57:19,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:19,799 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:19,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:57:19,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:19,800 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:57:19,800 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:57:19,800 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:57:19,800 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:57:19,800 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:19,800 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:57:19,800 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:57:19,800 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:57:19,800 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. 2023-03-21 11:57:19,800 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. started... 2023-03-21 11:57:19,800 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c7c660a43107606cd80c805b0450f72f 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-03-21 11:57:19,800 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c7c660a43107606cd80c805b0450f72f, store=A 2023-03-21 11:57:19,801 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:57:19,801 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c7c660a43107606cd80c805b0450f72f, store=B 2023-03-21 11:57:19,801 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:57:19,801 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c7c660a43107606cd80c805b0450f72f, store=C 2023-03-21 11:57:19,801 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:57:20,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(9158): Flush requested on c7c660a43107606cd80c805b0450f72f 2023-03-21 11:57:20,019 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. as already flushing 2023-03-21 11:57:20,040 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:20,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41068 deadline: 1679399900033, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:20,040 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:20,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41034 deadline: 1679399900033, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:20,040 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:20,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41010 deadline: 1679399900035, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:20,042 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:20,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41056 deadline: 1679399900038, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:20,044 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:20,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41084 deadline: 1679399900039, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:20,144 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:20,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41068 deadline: 1679399900141, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:20,144 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:20,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41034 deadline: 1679399900141, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:20,145 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:20,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41010 deadline: 1679399900141, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:20,146 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:20,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41056 deadline: 1679399900142, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:20,151 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:20,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41084 deadline: 1679399900145, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:20,213 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032131cecce316084f0e8f4b215d8d017733 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032131cecce316084f0e8f4b215d8d017733 2023-03-21 11:57:20,349 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:20,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41068 deadline: 1679399900345, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:20,351 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:20,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41034 deadline: 1679399900346, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:20,351 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:20,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41010 deadline: 1679399900346, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:20,351 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:20,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41056 deadline: 1679399900347, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:20,357 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:20,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41084 deadline: 1679399900352, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:20,617 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=120, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/.tmp/A/d902305aaee54d46ad041bf60b640a52 2023-03-21 11:57:20,653 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:20,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41068 deadline: 1679399900652, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:20,653 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:20,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41010 deadline: 1679399900652, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:20,654 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:20,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41056 deadline: 1679399900652, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:20,658 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:20,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41034 deadline: 1679399900654, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:20,660 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:20,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41084 deadline: 1679399900658, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:21,027 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=120 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/.tmp/B/a211ea3458014176b43db706fe04b0e4 2023-03-21 11:57:21,157 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:21,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41068 deadline: 1679399901154, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:21,157 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:21,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41010 deadline: 1679399901155, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:21,162 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:21,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41056 deadline: 1679399901157, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:21,163 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:21,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41034 deadline: 1679399901159, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:21,163 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:21,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41084 deadline: 1679399901161, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:21,435 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=120 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/.tmp/C/1ee84e90c2e64473ae96854d55e2625a 2023-03-21 11:57:21,438 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/.tmp/A/d902305aaee54d46ad041bf60b640a52 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/A/d902305aaee54d46ad041bf60b640a52 2023-03-21 11:57:21,440 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/A/d902305aaee54d46ad041bf60b640a52, entries=150, sequenceid=120, filesize=25.1 K 2023-03-21 11:57:21,441 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/.tmp/B/a211ea3458014176b43db706fe04b0e4 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/B/a211ea3458014176b43db706fe04b0e4 2023-03-21 11:57:21,444 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/B/a211ea3458014176b43db706fe04b0e4, entries=150, sequenceid=120, filesize=11.6 K 2023-03-21 11:57:21,444 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/.tmp/C/1ee84e90c2e64473ae96854d55e2625a as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/C/1ee84e90c2e64473ae96854d55e2625a 2023-03-21 11:57:21,447 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/C/1ee84e90c2e64473ae96854d55e2625a, entries=150, sequenceid=120, filesize=11.6 K 2023-03-21 11:57:21,448 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for c7c660a43107606cd80c805b0450f72f in 1648ms, sequenceid=120, compaction requested=true 2023-03-21 11:57:21,448 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c7c660a43107606cd80c805b0450f72f: 2023-03-21 11:57:21,448 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. 2023-03-21 11:57:21,448 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:57:21,448 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:57:21,448 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:21,448 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:57:21,448 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:57:21,449 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:21,449 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:21,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:21,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:21,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:21,449 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:21,449 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:57:21,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:21,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:21,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:21,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:21,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:21,450 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:57:21,450 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@371a950c[Count = 0] remaining members to acquire global barrier 2023-03-21 11:57:21,450 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:57:21,450 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:21,451 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:21,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:21,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:21,451 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:57:21,451 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:57:21,451 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:57:21,451 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:21,451 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:57:21,452 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:21,452 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:57:21,452 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:21,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:21,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:21,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:21,452 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:57:21,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:21,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:21,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:21,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:21,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:21,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:21,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:21,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:57:21,453 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:57:21,453 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:57:21,453 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:57:21,453 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:57:21,453 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:57:21,453 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:57:21,454 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:21,454 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:21,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:21,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:21,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:21,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:21,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:21,454 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:21,454 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:21,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:21,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:21,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:21,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:21,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:21,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:21,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:21,455 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:21,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:21,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:21,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:21,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:21,456 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:21,456 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:21,456 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:21,456 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:21,456 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:21,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:21,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:21,456 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:57:21,456 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:57:21,456 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:57:21,456 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:21,457 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:21,457 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:21,457 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:21,457 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:21,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:21,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:21,457 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:57:21,457 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:57:21,557 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:21,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:21,559 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:57:21,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:57:21,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:57:21,560 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:57:21,560 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:21,560 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:57:21,560 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:57:21,560 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:21,560 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:21,561 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:21,561 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:21,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:21,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:21,561 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:21,561 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:57:21,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:21,561 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:21,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:57:21,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:21,562 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:57:21,562 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:57:21,562 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:57:21,562 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:57:21,562 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:21,562 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:57:21,562 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:57:21,562 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:57:21,562 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. 2023-03-21 11:57:21,562 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. started... 2023-03-21 11:57:21,562 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c7c660a43107606cd80c805b0450f72f 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-03-21 11:57:21,562 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c7c660a43107606cd80c805b0450f72f, store=A 2023-03-21 11:57:21,563 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:57:21,563 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c7c660a43107606cd80c805b0450f72f, store=B 2023-03-21 11:57:21,563 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:57:21,563 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c7c660a43107606cd80c805b0450f72f, store=C 2023-03-21 11:57:21,563 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:57:21,979 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230321fd673f7ecc1f46d7a0f38f3aa4977880 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230321fd673f7ecc1f46d7a0f38f3aa4977880 2023-03-21 11:57:22,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(9158): Flush requested on c7c660a43107606cd80c805b0450f72f 2023-03-21 11:57:22,163 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. as already flushing 2023-03-21 11:57:22,193 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:22,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41084 deadline: 1679399902181, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:22,196 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:22,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41056 deadline: 1679399902188, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:22,196 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:22,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41068 deadline: 1679399902188, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:22,196 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:22,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41034 deadline: 1679399902189, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:22,196 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:22,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41010 deadline: 1679399902191, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:22,295 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:22,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41084 deadline: 1679399902294, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:22,301 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:22,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41056 deadline: 1679399902297, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:22,301 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:22,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41068 deadline: 1679399902297, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:22,302 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:22,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41034 deadline: 1679399902297, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:22,302 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:22,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41010 deadline: 1679399902297, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:22,383 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=135, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/.tmp/A/85d8dd2b89324fcebc7c831cc96071c9 2023-03-21 11:57:22,502 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:22,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41084 deadline: 1679399902497, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:22,504 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:22,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41056 deadline: 1679399902502, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:22,504 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:22,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41068 deadline: 1679399902502, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:22,504 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:22,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41034 deadline: 1679399902503, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:22,504 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:22,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41010 deadline: 1679399902503, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:22,791 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=135 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/.tmp/B/55de6e2feb69480ab42f0660aaf24831 2023-03-21 11:57:22,802 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=135 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/.tmp/C/0e2f9e6c7e9b4c6486bb24b96c320049 2023-03-21 11:57:22,806 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/.tmp/A/85d8dd2b89324fcebc7c831cc96071c9 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/A/85d8dd2b89324fcebc7c831cc96071c9 2023-03-21 11:57:22,809 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/A/85d8dd2b89324fcebc7c831cc96071c9, entries=150, sequenceid=135, filesize=25.3 K 2023-03-21 11:57:22,810 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/.tmp/B/55de6e2feb69480ab42f0660aaf24831 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/B/55de6e2feb69480ab42f0660aaf24831 2023-03-21 11:57:22,811 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:22,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41084 deadline: 1679399902805, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:22,812 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:22,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41068 deadline: 1679399902805, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:22,813 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/B/55de6e2feb69480ab42f0660aaf24831, entries=150, sequenceid=135, filesize=11.8 K 2023-03-21 11:57:22,813 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:22,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41010 deadline: 1679399902806, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:22,813 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:22,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41056 deadline: 1679399902807, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:22,813 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:22,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41034 deadline: 1679399902807, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:22,814 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/.tmp/C/0e2f9e6c7e9b4c6486bb24b96c320049 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/C/0e2f9e6c7e9b4c6486bb24b96c320049 2023-03-21 11:57:22,816 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/C/0e2f9e6c7e9b4c6486bb24b96c320049, entries=150, sequenceid=135, filesize=11.8 K 2023-03-21 11:57:22,817 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for c7c660a43107606cd80c805b0450f72f in 1255ms, sequenceid=135, compaction requested=true 2023-03-21 11:57:22,817 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c7c660a43107606cd80c805b0450f72f: 2023-03-21 11:57:22,817 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. 2023-03-21 11:57:22,817 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:57:22,817 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:57:22,817 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:22,817 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:57:22,817 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:57:22,818 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:22,818 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:22,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:22,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:22,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:22,819 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:22,819 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:57:22,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:22,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:22,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:22,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:22,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:22,820 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:57:22,820 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@378a7b31[Count = 0] remaining members to acquire global barrier 2023-03-21 11:57:22,820 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:57:22,820 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:22,821 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:22,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:22,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:22,821 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:57:22,821 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:57:22,821 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:57:22,821 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:22,821 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:57:22,822 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:57:22,822 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:22,822 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:22,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:22,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:22,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:22,822 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:57:22,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:22,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:22,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:22,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:22,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:22,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:22,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:22,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:57:22,824 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:57:22,824 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:57:22,824 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:57:22,824 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:57:22,824 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:57:22,824 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:57:22,825 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:22,825 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:22,825 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:22,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:22,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:22,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:22,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:22,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:22,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:22,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:22,826 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:22,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:22,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:22,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:22,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:22,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:22,827 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:22,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:22,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:22,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:22,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:22,829 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:22,829 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:22,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:22,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:22,829 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:22,829 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:57:22,829 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:57:22,829 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:57:22,830 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:22,830 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:57:22,830 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:57:22,830 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:22,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:22,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:22,830 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:22,830 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:22,830 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:22,830 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:22,830 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:22,930 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:22,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:22,932 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:57:22,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:57:22,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:57:22,933 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:57:22,933 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:22,933 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:57:22,933 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:57:22,933 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:22,933 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:22,934 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:22,934 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:22,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:22,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:22,934 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:22,934 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:57:22,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:22,934 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:22,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:57:22,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:22,935 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:57:22,935 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:57:22,935 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:57:22,935 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:57:22,935 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:22,935 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:57:22,936 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:57:22,936 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:57:22,936 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. 2023-03-21 11:57:22,936 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. started... 2023-03-21 11:57:22,936 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c7c660a43107606cd80c805b0450f72f 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-03-21 11:57:22,936 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c7c660a43107606cd80c805b0450f72f, store=A 2023-03-21 11:57:22,936 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:57:22,936 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c7c660a43107606cd80c805b0450f72f, store=B 2023-03-21 11:57:22,936 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:57:22,936 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c7c660a43107606cd80c805b0450f72f, store=C 2023-03-21 11:57:22,936 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:57:22,946 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230321350913b8a56c4b87bbb7f45b69c730c8 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230321350913b8a56c4b87bbb7f45b69c730c8 2023-03-21 11:57:23,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(9158): Flush requested on c7c660a43107606cd80c805b0450f72f 2023-03-21 11:57:23,316 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. as already flushing 2023-03-21 11:57:23,337 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:23,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41010 deadline: 1679399903328, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:23,337 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:23,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41034 deadline: 1679399903328, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:23,337 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:23,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41084 deadline: 1679399903329, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:23,338 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:23,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41056 deadline: 1679399903331, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:23,338 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:23,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41068 deadline: 1679399903333, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:23,350 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=152, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/.tmp/A/01282560f5c44eaaa13fa7ef2aa51c63 2023-03-21 11:57:23,444 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:23,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41034 deadline: 1679399903438, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:23,444 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:23,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41010 deadline: 1679399903438, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:23,444 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:23,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41084 deadline: 1679399903438, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:23,444 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:23,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41056 deadline: 1679399903438, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:23,445 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:23,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41068 deadline: 1679399903439, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:23,648 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:23,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41034 deadline: 1679399903645, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:23,649 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:23,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41010 deadline: 1679399903645, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:23,649 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:23,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41084 deadline: 1679399903645, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:23,649 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:23,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41056 deadline: 1679399903645, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:23,649 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:23,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41068 deadline: 1679399903646, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:23,757 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=152 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/.tmp/B/48abbb6bd42b4d5796b7537472f0b3b0 2023-03-21 11:57:23,952 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:23,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41010 deadline: 1679399903950, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:23,952 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:23,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41068 deadline: 1679399903950, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:23,953 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:23,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41084 deadline: 1679399903951, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:23,953 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:23,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41056 deadline: 1679399903951, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:23,957 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:23,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41034 deadline: 1679399903951, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:24,167 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=152 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/.tmp/C/b35a675ef3f44edcac89a939f27ca34e 2023-03-21 11:57:24,170 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/.tmp/A/01282560f5c44eaaa13fa7ef2aa51c63 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/A/01282560f5c44eaaa13fa7ef2aa51c63 2023-03-21 11:57:24,173 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/A/01282560f5c44eaaa13fa7ef2aa51c63, entries=150, sequenceid=152, filesize=25.3 K 2023-03-21 11:57:24,174 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/.tmp/B/48abbb6bd42b4d5796b7537472f0b3b0 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/B/48abbb6bd42b4d5796b7537472f0b3b0 2023-03-21 11:57:24,177 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/B/48abbb6bd42b4d5796b7537472f0b3b0, entries=150, sequenceid=152, filesize=11.8 K 2023-03-21 11:57:24,177 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/.tmp/C/b35a675ef3f44edcac89a939f27ca34e as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/C/b35a675ef3f44edcac89a939f27ca34e 2023-03-21 11:57:24,180 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/C/b35a675ef3f44edcac89a939f27ca34e, entries=150, sequenceid=152, filesize=11.8 K 2023-03-21 11:57:24,180 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for c7c660a43107606cd80c805b0450f72f in 1244ms, sequenceid=152, compaction requested=true 2023-03-21 11:57:24,180 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c7c660a43107606cd80c805b0450f72f: 2023-03-21 11:57:24,180 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. 2023-03-21 11:57:24,180 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:57:24,181 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:57:24,181 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:24,181 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:57:24,181 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:57:24,181 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:24,181 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:24,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:24,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:24,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:24,182 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:24,182 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:57:24,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:24,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:24,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:24,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:24,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:24,183 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:57:24,183 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@53060114[Count = 0] remaining members to acquire global barrier 2023-03-21 11:57:24,183 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:57:24,183 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:24,184 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:24,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:24,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:24,184 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:57:24,184 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:57:24,184 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:57:24,184 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:24,184 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:57:24,185 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:57:24,185 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:24,185 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:24,185 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:57:24,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:24,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:24,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:24,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:24,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:24,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:24,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:24,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:24,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:24,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:24,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:57:24,187 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:57:24,187 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:57:24,187 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:57:24,187 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:57:24,187 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:57:24,187 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:57:24,188 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:24,188 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:24,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:24,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:24,188 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:24,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:24,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:24,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:24,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:24,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:24,188 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:24,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:24,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:24,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:24,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:24,189 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:24,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:24,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:24,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:24,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:24,190 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:24,190 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:24,191 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:24,191 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:24,191 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:57:24,191 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:57:24,191 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:24,191 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:24,191 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:24,191 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:57:24,191 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:57:24,191 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:24,191 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:57:24,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:24,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:24,191 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:24,191 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:24,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:24,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:24,291 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:24,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:24,293 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:57:24,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:57:24,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:57:24,294 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:57:24,294 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:24,294 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:57:24,294 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:57:24,294 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:24,294 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:24,295 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:24,295 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:24,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:24,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:24,295 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:24,295 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:57:24,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:24,296 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:24,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:57:24,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:24,296 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:57:24,296 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:57:24,296 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:57:24,296 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:57:24,296 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:24,296 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:57:24,296 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:57:24,296 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:57:24,296 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. 2023-03-21 11:57:24,296 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. started... 2023-03-21 11:57:24,297 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c7c660a43107606cd80c805b0450f72f 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-03-21 11:57:24,297 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c7c660a43107606cd80c805b0450f72f, store=A 2023-03-21 11:57:24,297 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:57:24,297 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c7c660a43107606cd80c805b0450f72f, store=B 2023-03-21 11:57:24,297 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:57:24,297 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c7c660a43107606cd80c805b0450f72f, store=C 2023-03-21 11:57:24,297 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:57:24,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(9158): Flush requested on c7c660a43107606cd80c805b0450f72f 2023-03-21 11:57:24,460 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. as already flushing 2023-03-21 11:57:24,494 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:24,494 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:24,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41010 deadline: 1679399904486, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:24,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41084 deadline: 1679399904486, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:24,495 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:24,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41056 deadline: 1679399904486, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:24,495 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:24,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41068 deadline: 1679399904487, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:24,495 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:24,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41034 deadline: 1679399904487, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:24,599 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:24,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41084 deadline: 1679399904595, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:24,599 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:24,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41010 deadline: 1679399904595, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:24,599 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:24,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41056 deadline: 1679399904596, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:24,599 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:24,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41068 deadline: 1679399904596, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:24,599 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:24,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41034 deadline: 1679399904596, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:24,707 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032103ac55eb5ddb476d962a75690363d242 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032103ac55eb5ddb476d962a75690363d242 2023-03-21 11:57:24,803 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:24,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41084 deadline: 1679399904800, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:24,804 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:24,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41068 deadline: 1679399904800, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:24,804 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:24,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41010 deadline: 1679399904801, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:24,804 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:24,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41056 deadline: 1679399904801, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:24,804 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:24,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41034 deadline: 1679399904801, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:25,111 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:25,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41084 deadline: 1679399905104, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:25,112 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:25,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41068 deadline: 1679399905105, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:25,112 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:25,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41010 deadline: 1679399905106, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:25,113 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:25,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41056 deadline: 1679399905107, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:25,113 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:25,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41034 deadline: 1679399905107, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:25,114 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=167, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/.tmp/A/9a13801312144409a5626ba50cea90a9 2023-03-21 11:57:25,521 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=167 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/.tmp/B/113525b06f8742068b3bc866ee2e4eee 2023-03-21 11:57:25,618 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:25,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41068 deadline: 1679399905614, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:25,618 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:25,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41010 deadline: 1679399905616, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:25,618 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:25,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41084 deadline: 1679399905616, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:25,618 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:25,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41056 deadline: 1679399905616, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:25,618 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:25,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41034 deadline: 1679399905616, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:25,930 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=167 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/.tmp/C/14a2181438834a78bbfc31c56abe9522 2023-03-21 11:57:25,933 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/.tmp/A/9a13801312144409a5626ba50cea90a9 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/A/9a13801312144409a5626ba50cea90a9 2023-03-21 11:57:25,936 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/A/9a13801312144409a5626ba50cea90a9, entries=150, sequenceid=167, filesize=25.3 K 2023-03-21 11:57:25,937 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/.tmp/B/113525b06f8742068b3bc866ee2e4eee as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/B/113525b06f8742068b3bc866ee2e4eee 2023-03-21 11:57:25,940 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/B/113525b06f8742068b3bc866ee2e4eee, entries=150, sequenceid=167, filesize=11.8 K 2023-03-21 11:57:25,940 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/.tmp/C/14a2181438834a78bbfc31c56abe9522 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/C/14a2181438834a78bbfc31c56abe9522 2023-03-21 11:57:25,943 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/C/14a2181438834a78bbfc31c56abe9522, entries=150, sequenceid=167, filesize=11.8 K 2023-03-21 11:57:25,943 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=100.63 KB/103050 for c7c660a43107606cd80c805b0450f72f in 1647ms, sequenceid=167, compaction requested=true 2023-03-21 11:57:25,943 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c7c660a43107606cd80c805b0450f72f: 2023-03-21 11:57:25,943 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. 2023-03-21 11:57:25,943 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:57:25,944 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:57:25,944 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:25,944 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:57:25,944 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:57:25,945 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:25,945 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:25,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:25,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:25,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:25,945 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:25,945 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:57:25,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:25,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:25,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:25,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:25,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:25,946 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:57:25,946 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7910e8d4[Count = 0] remaining members to acquire global barrier 2023-03-21 11:57:25,946 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:57:25,946 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:25,946 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:25,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:25,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:25,946 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:57:25,946 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:57:25,946 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:25,946 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:57:25,946 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:57:25,947 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:57:25,947 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:25,947 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:25,947 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:57:25,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:25,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:25,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:25,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:25,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:25,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:25,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:25,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:25,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:25,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:25,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:57:25,949 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:57:25,949 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:57:25,949 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:57:25,949 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:57:25,949 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:57:25,949 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:57:25,950 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:25,950 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:25,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:25,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:25,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:25,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:25,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:25,950 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:25,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:25,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:25,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:25,950 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:25,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:25,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:25,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:25,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:25,951 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:25,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:25,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:25,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:25,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:25,952 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:25,952 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:25,952 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:25,952 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:57:25,952 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:57:25,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:25,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:25,952 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:25,952 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:57:25,953 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:57:25,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:25,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:25,952 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:57:25,952 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:25,952 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:25,953 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:25,953 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:25,953 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:25,953 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:26,053 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:26,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:26,054 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:57:26,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:57:26,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:57:26,055 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:57:26,056 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:26,056 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:57:26,056 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:57:26,056 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:26,056 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:26,056 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:26,056 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:26,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:26,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:26,057 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:26,057 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:57:26,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:26,057 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:26,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:57:26,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:26,057 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:57:26,057 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:57:26,057 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:57:26,057 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:57:26,057 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:26,058 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:57:26,058 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:57:26,058 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:57:26,058 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. 2023-03-21 11:57:26,058 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. started... 2023-03-21 11:57:26,058 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c7c660a43107606cd80c805b0450f72f 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-03-21 11:57:26,058 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c7c660a43107606cd80c805b0450f72f, store=A 2023-03-21 11:57:26,058 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:57:26,058 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c7c660a43107606cd80c805b0450f72f, store=B 2023-03-21 11:57:26,058 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:57:26,058 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c7c660a43107606cd80c805b0450f72f, store=C 2023-03-21 11:57:26,058 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:57:26,470 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303214efa2554ab534decaa200f37fe54a016 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303214efa2554ab534decaa200f37fe54a016 2023-03-21 11:57:26,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(9158): Flush requested on c7c660a43107606cd80c805b0450f72f 2023-03-21 11:57:26,624 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. as already flushing 2023-03-21 11:57:26,646 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:26,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41068 deadline: 1679399906640, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:26,649 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:26,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41034 deadline: 1679399906645, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:26,649 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:26,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41010 deadline: 1679399906645, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:26,655 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:26,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41056 deadline: 1679399906646, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:26,655 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:26,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41084 deadline: 1679399906646, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:26,750 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:26,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41068 deadline: 1679399906747, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:26,757 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:26,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41034 deadline: 1679399906750, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:26,757 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:26,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41010 deadline: 1679399906750, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:26,758 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:26,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41056 deadline: 1679399906756, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:26,759 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:26,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41084 deadline: 1679399906756, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:26,876 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=185, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/.tmp/A/bb9922a3268644ad819a55cf27aaabb0 2023-03-21 11:57:26,886 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=185 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/.tmp/B/6f6a7ab02ea443d8a9686845d519df44 2023-03-21 11:57:26,894 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=185 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/.tmp/C/5dcf26970f6a4201a8d85a9190acb7e1 2023-03-21 11:57:26,897 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/.tmp/A/bb9922a3268644ad819a55cf27aaabb0 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/A/bb9922a3268644ad819a55cf27aaabb0 2023-03-21 11:57:26,900 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/A/bb9922a3268644ad819a55cf27aaabb0, entries=150, sequenceid=185, filesize=25.3 K 2023-03-21 11:57:26,901 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/.tmp/B/6f6a7ab02ea443d8a9686845d519df44 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/B/6f6a7ab02ea443d8a9686845d519df44 2023-03-21 11:57:26,904 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/B/6f6a7ab02ea443d8a9686845d519df44, entries=150, sequenceid=185, filesize=11.8 K 2023-03-21 11:57:26,904 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/.tmp/C/5dcf26970f6a4201a8d85a9190acb7e1 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/C/5dcf26970f6a4201a8d85a9190acb7e1 2023-03-21 11:57:26,908 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/C/5dcf26970f6a4201a8d85a9190acb7e1, entries=150, sequenceid=185, filesize=11.8 K 2023-03-21 11:57:26,909 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=80.51 KB/82440 for c7c660a43107606cd80c805b0450f72f in 851ms, sequenceid=185, compaction requested=true 2023-03-21 11:57:26,909 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c7c660a43107606cd80c805b0450f72f: 2023-03-21 11:57:26,909 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. 2023-03-21 11:57:26,909 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:57:26,909 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:57:26,909 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:26,909 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:57:26,909 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:57:26,910 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:26,910 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:26,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:26,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:26,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:26,910 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:26,910 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:57:26,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:26,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:26,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:26,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:26,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:26,911 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:57:26,911 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1ac1a00f[Count = 0] remaining members to acquire global barrier 2023-03-21 11:57:26,911 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:57:26,911 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:26,912 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:26,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:26,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:26,912 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:26,912 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:57:26,912 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:57:26,912 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:57:26,912 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:57:26,913 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:57:26,913 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:26,913 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:26,913 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:57:26,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:26,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:26,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:26,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:26,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:26,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:26,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:26,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:26,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:26,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:26,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:57:26,915 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:57:26,915 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:57:26,915 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:57:26,915 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:57:26,915 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:57:26,915 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:57:26,916 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:26,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:26,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:26,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:26,916 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:26,916 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:26,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:26,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:26,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:26,916 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:26,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:26,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:26,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:26,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:26,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:26,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:26,918 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:26,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:26,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:26,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:26,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:26,921 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:26,921 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:26,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:26,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:26,921 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:26,921 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:57:26,921 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:57:26,921 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:57:26,921 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:26,921 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:26,921 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:26,921 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:26,921 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:26,921 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:57:26,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:26,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:26,921 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:26,921 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:57:26,922 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:26,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(9158): Flush requested on c7c660a43107606cd80c805b0450f72f 2023-03-21 11:57:26,957 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c7c660a43107606cd80c805b0450f72f 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-03-21 11:57:26,958 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c7c660a43107606cd80c805b0450f72f, store=A 2023-03-21 11:57:26,958 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:57:26,958 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c7c660a43107606cd80c805b0450f72f, store=B 2023-03-21 11:57:26,958 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:57:26,958 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c7c660a43107606cd80c805b0450f72f, store=C 2023-03-21 11:57:26,958 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:57:26,996 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:26,996 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:26,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41056 deadline: 1679399906985, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:26,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41084 deadline: 1679399906986, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,003 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:27,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41034 deadline: 1679399906994, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,005 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:27,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41068 deadline: 1679399906995, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,005 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:27,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41010 deadline: 1679399906996, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,022 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:27,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:27,023 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:57:27,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:57:27,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:57:27,025 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:57:27,025 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:27,025 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:57:27,025 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:57:27,025 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:27,025 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:27,026 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:27,026 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:27,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:27,026 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,026 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:57:27,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:27,027 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:27,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:57:27,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:27,027 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:57:27,027 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:57:27,027 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:57:27,027 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:57:27,027 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:27,027 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:57:27,028 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:57:27,028 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:57:27,028 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. 2023-03-21 11:57:27,028 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. started... 2023-03-21 11:57:27,028 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. as already flushing 2023-03-21 11:57:27,028 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. 2023-03-21 11:57:27,028 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:57:27,028 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:57:27,028 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,028 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:57:27,028 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:57:27,029 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,029 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:27,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:27,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:27,029 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:27,029 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:57:27,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:27,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:27,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:27,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:27,030 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:57:27,030 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:57:27,030 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ef60395[Count = 0] remaining members to acquire global barrier 2023-03-21 11:57:27,030 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:27,031 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:27,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:27,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:27,031 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:57:27,031 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,031 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:57:27,031 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:57:27,031 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:57:27,032 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,032 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:57:27,032 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:27,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:27,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:27,032 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:57:27,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:27,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:27,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:27,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:27,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:27,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:57:27,034 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:57:27,034 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:57:27,034 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:57:27,034 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:57:27,034 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:57:27,034 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:57:27,035 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:27,035 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:27,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:27,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:27,036 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:27,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:27,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:27,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:27,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:27,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:27,036 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:27,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:27,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:27,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:27,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:27,037 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:27,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:27,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,038 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:27,038 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:27,038 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:27,039 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:27,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:27,039 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:27,039 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:27,039 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,039 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:27,039 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:27,039 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:27,039 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:57:27,039 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:57:27,039 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:57:27,039 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:57:27,039 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:57:27,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:27,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:27,102 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:27,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41084 deadline: 1679399907097, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,103 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:27,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41056 deadline: 1679399907097, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,120 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:27,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41034 deadline: 1679399907104, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,120 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:27,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41068 deadline: 1679399907106, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,120 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:27,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41010 deadline: 1679399907106, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,139 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:27,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:27,141 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:57:27,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:57:27,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:57:27,142 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:57:27,142 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:27,142 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:57:27,142 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:57:27,142 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:27,142 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:27,143 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:27,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:27,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:27,143 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:27,144 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,144 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:57:27,144 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:27,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:57:27,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:27,144 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:57:27,144 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:57:27,144 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:57:27,144 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:57:27,144 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:27,144 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:57:27,145 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:57:27,145 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:57:27,145 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. 2023-03-21 11:57:27,145 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. started... 2023-03-21 11:57:27,145 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. as already flushing 2023-03-21 11:57:27,145 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. 2023-03-21 11:57:27,145 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:57:27,145 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:57:27,145 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,145 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:57:27,145 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:57:27,146 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,146 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:27,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:27,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:27,146 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:27,146 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:57:27,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:27,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:27,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:27,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:27,147 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:57:27,147 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d1ba3fe[Count = 0] remaining members to acquire global barrier 2023-03-21 11:57:27,147 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:57:27,147 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:27,148 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:27,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:27,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:27,148 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:57:27,148 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:57:27,148 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,148 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:57:27,148 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:57:27,148 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,148 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:57:27,148 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:27,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:27,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:27,149 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:57:27,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:27,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:27,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:27,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:27,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:27,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:57:27,150 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:57:27,150 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:57:27,150 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:57:27,150 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:57:27,150 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:57:27,150 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:57:27,151 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:27,151 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:27,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:27,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:27,151 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:27,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:27,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:27,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:27,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:27,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:27,151 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:27,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:27,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:27,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:27,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:27,152 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:27,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:27,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,153 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:27,153 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:27,154 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:57:27,154 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:57:27,153 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:27,154 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:57:27,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:27,154 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:27,154 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:27,154 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:27,154 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,154 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:27,154 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:57:27,154 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:27,154 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:57:27,154 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:27,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:27,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:27,254 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:27,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:27,256 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:57:27,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:57:27,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:57:27,260 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:57:27,260 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:27,260 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:57:27,260 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:57:27,261 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:27,261 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:27,261 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:27,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:27,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:27,262 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:27,262 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,262 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:57:27,262 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:27,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:57:27,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:27,262 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:57:27,262 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:57:27,263 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:57:27,263 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:57:27,263 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:27,263 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:57:27,263 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:57:27,263 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:57:27,263 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. 2023-03-21 11:57:27,263 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. started... 2023-03-21 11:57:27,263 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. as already flushing 2023-03-21 11:57:27,263 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. 2023-03-21 11:57:27,263 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:57:27,263 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:57:27,263 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,263 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:57:27,263 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:57:27,264 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,264 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:27,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:27,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:27,264 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:27,264 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:57:27,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:27,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:27,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:27,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:27,265 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:57:27,265 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@63b3aec1[Count = 0] remaining members to acquire global barrier 2023-03-21 11:57:27,265 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:57:27,265 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:27,266 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:27,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:27,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:27,266 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:57:27,266 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:57:27,266 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:57:27,266 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,266 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:57:27,266 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:57:27,266 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:27,266 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,266 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:57:27,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:27,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:27,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:27,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:27,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:27,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:27,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:27,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:57:27,268 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:57:27,268 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:57:27,268 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:57:27,268 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:57:27,268 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:57:27,268 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:57:27,269 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:27,269 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:27,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:27,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:27,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:27,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:27,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:27,269 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:27,269 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:27,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:27,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:27,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:27,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:27,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:27,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:27,270 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:27,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:27,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,271 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,271 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:27,271 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:27,271 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:27,271 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:57:27,271 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:57:27,271 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:27,272 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,272 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:27,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:27,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:27,272 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:57:27,272 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:57:27,271 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:27,272 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:57:27,272 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:27,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:27,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:27,272 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:27,312 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:27,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41084 deadline: 1679399907304, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,312 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:27,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41056 deadline: 1679399907304, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,323 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:27,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41034 deadline: 1679399907321, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,323 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:27,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41010 deadline: 1679399907321, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,323 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:27,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41068 deadline: 1679399907322, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,370 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032179cf71e5d33749baa5cc182e1644a8df to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032179cf71e5d33749baa5cc182e1644a8df 2023-03-21 11:57:27,372 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:27,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:27,374 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:57:27,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:57:27,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:57:27,376 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:57:27,376 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:27,377 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:57:27,377 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:57:27,377 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:27,377 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:27,377 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,378 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:27,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:27,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:27,378 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,378 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:57:27,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:27,378 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:27,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:57:27,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:27,378 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:57:27,378 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:57:27,378 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:57:27,378 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:57:27,379 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:27,379 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:57:27,386 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:57:27,386 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. 2023-03-21 11:57:27,386 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:57:27,386 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. started... 2023-03-21 11:57:27,386 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. as already flushing 2023-03-21 11:57:27,386 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. 2023-03-21 11:57:27,386 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:57:27,386 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:57:27,386 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,386 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:57:27,386 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:57:27,394 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:27,394 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:27,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:27,395 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:27,395 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:57:27,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:27,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:27,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:27,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:27,396 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:57:27,396 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7465bb60[Count = 0] remaining members to acquire global barrier 2023-03-21 11:57:27,396 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:57:27,396 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:27,398 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:27,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:27,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:27,398 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:57:27,398 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:57:27,398 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,398 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:57:27,398 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:57:27,399 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:57:27,399 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,399 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:27,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:27,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:27,399 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:57:27,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:27,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:27,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:27,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:27,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:27,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:57:27,401 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:57:27,401 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:57:27,401 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:57:27,401 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:57:27,401 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:57:27,401 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:57:27,404 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:27,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:27,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:27,404 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:27,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:27,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:27,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:27,404 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:27,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:27,404 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:27,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:27,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:27,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:27,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:27,405 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:27,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:27,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:27,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,407 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:27,407 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:27,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:27,407 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:27,407 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:57:27,407 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:57:27,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:27,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:27,407 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:27,407 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:27,407 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:57:27,407 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:57:27,407 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:57:27,407 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:27,408 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,408 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:27,408 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:27,408 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:27,508 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:27,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:27,509 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:57:27,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:57:27,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:57:27,510 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:57:27,510 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:27,510 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:57:27,510 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:57:27,510 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:27,511 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:27,511 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:27,511 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:27,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:27,511 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,511 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:57:27,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:27,512 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:27,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:57:27,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:27,512 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:57:27,512 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:57:27,512 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:57:27,512 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:57:27,512 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:27,512 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:57:27,512 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:57:27,512 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:57:27,512 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. 2023-03-21 11:57:27,513 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. started... 2023-03-21 11:57:27,513 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. as already flushing 2023-03-21 11:57:27,513 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. 2023-03-21 11:57:27,513 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:57:27,513 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:57:27,513 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,513 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:57:27,513 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:57:27,513 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,514 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:27,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:27,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:27,514 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:27,514 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:57:27,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:27,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:27,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:27,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:27,515 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:57:27,515 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3a41f4d2[Count = 0] remaining members to acquire global barrier 2023-03-21 11:57:27,515 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:57:27,515 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:27,515 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:27,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:27,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:27,515 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:57:27,515 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:57:27,515 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,516 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:57:27,515 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:57:27,516 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:27,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:27,516 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:57:27,516 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:27,516 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:57:27,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:27,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:27,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:27,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:27,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:27,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:57:27,518 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:57:27,518 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:57:27,518 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:57:27,518 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:57:27,518 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:57:27,518 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:57:27,519 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:27,519 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:27,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:27,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:27,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:27,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:27,519 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:27,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:27,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:27,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:27,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:27,520 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:27,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:27,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:27,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:27,521 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:27,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:27,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,522 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,522 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:27,522 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:27,523 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:27,523 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:57:27,523 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:57:27,523 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:27,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:27,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:27,523 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:27,523 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:57:27,523 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:57:27,523 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,523 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:27,523 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:27,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:27,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:27,523 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:27,618 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:27,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41056 deadline: 1679399907613, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,618 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:27,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41084 deadline: 1679399907614, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,623 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:27,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:27,625 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:57:27,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:57:27,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:57:27,626 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:57:27,626 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:27,626 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:57:27,626 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:57:27,626 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:27,626 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:27,627 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,627 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:27,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:27,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:27,627 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,627 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:57:27,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:27,627 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:27,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:57:27,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:27,628 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:57:27,628 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:57:27,628 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:57:27,628 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:57:27,628 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:27,628 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:27,628 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:57:27,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41068 deadline: 1679399907625, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,628 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:57:27,628 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:57:27,628 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:27,628 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. 2023-03-21 11:57:27,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41034 deadline: 1679399907626, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,629 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. started... 2023-03-21 11:57:27,629 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:27,629 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. as already flushing 2023-03-21 11:57:27,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41010 deadline: 1679399907626, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,629 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. 2023-03-21 11:57:27,629 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:57:27,629 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:57:27,629 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,629 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:57:27,629 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:57:27,630 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,630 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:27,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:27,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:27,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:27,630 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:27,630 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:57:27,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:27,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:27,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:27,631 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:57:27,631 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a05b437[Count = 0] remaining members to acquire global barrier 2023-03-21 11:57:27,631 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:57:27,631 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:27,632 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:27,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:27,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:27,632 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,632 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:57:27,632 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:57:27,632 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:57:27,632 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:57:27,633 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,633 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:57:27,633 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:27,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:27,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:27,633 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:57:27,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:27,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:27,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:27,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:27,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:27,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:57:27,635 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:57:27,635 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:57:27,635 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:57:27,635 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:57:27,635 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:57:27,635 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:57:27,636 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:27,636 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:27,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:27,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:27,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:27,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:27,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:27,636 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:27,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:27,636 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:27,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:27,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:27,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:27,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:27,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:27,637 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:27,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:27,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,638 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,638 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:27,638 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:27,638 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:27,638 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:57:27,638 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:57:27,638 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:27,638 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:27,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:27,638 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:27,638 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:57:27,638 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:27,639 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:57:27,639 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:57:27,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:27,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:27,639 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:27,639 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:27,739 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:27,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:27,741 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:57:27,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:57:27,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:57:27,741 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:57:27,741 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:27,741 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:57:27,741 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:57:27,743 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:27,743 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:27,743 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,743 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:27,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:27,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:27,744 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,744 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:57:27,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:27,744 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:27,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:57:27,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:27,744 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:57:27,744 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:57:27,744 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:57:27,745 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:57:27,745 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:27,745 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:57:27,745 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:57:27,745 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:57:27,745 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. 2023-03-21 11:57:27,745 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. started... 2023-03-21 11:57:27,745 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. as already flushing 2023-03-21 11:57:27,745 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. 2023-03-21 11:57:27,745 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:57:27,745 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:57:27,745 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,745 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:57:27,745 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:57:27,746 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:27,746 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:27,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:27,746 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:27,746 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:57:27,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:27,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:27,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:27,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:27,747 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:57:27,747 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@43ec810[Count = 0] remaining members to acquire global barrier 2023-03-21 11:57:27,747 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:57:27,747 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:27,747 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:27,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:27,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:27,748 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:57:27,748 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,748 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:57:27,748 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:57:27,748 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:57:27,748 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,748 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:57:27,748 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:27,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:27,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:27,749 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:57:27,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:27,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:27,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:27,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:27,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:27,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:57:27,750 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:57:27,750 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:57:27,750 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:57:27,751 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:57:27,751 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:57:27,751 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:57:27,752 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:27,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:27,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:27,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:27,752 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:27,752 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:27,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:27,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:27,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:27,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:27,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:27,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:27,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:27,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:27,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:27,753 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:27,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:27,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,754 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,756 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,756 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:27,756 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:27,756 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:27,756 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:57:27,756 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:57:27,756 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:27,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:27,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:27,756 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:27,756 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:57:27,756 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,756 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:57:27,756 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:27,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:27,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:27,756 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:27,756 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:27,776 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=201, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/.tmp/A/977fbb51bd5f4334a901ed7bcde18748 2023-03-21 11:57:27,856 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:27,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:27,858 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:57:27,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:57:27,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:57:27,859 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:57:27,859 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:27,859 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:57:27,859 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:57:27,859 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:27,859 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:27,860 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,860 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:27,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:27,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:27,860 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,860 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:57:27,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:27,861 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:27,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:57:27,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:27,861 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:57:27,861 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:57:27,861 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:57:27,861 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:57:27,861 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:27,861 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:57:27,861 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:57:27,861 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:57:27,861 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. 2023-03-21 11:57:27,861 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. started... 2023-03-21 11:57:27,862 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. as already flushing 2023-03-21 11:57:27,862 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. 2023-03-21 11:57:27,862 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:57:27,862 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:57:27,862 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,862 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:57:27,862 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:57:27,863 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,863 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:27,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:27,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:27,863 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:27,863 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:57:27,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:27,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:27,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:27,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:27,864 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:57:27,864 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:57:27,864 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5dd9cd4c[Count = 0] remaining members to acquire global barrier 2023-03-21 11:57:27,864 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:27,864 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:27,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:27,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:27,864 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:57:27,865 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:57:27,865 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,865 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:57:27,865 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:57:27,865 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:57:27,865 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:27,865 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,865 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:57:27,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:27,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:27,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:27,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:27,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:27,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:27,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:27,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:57:27,867 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:57:27,867 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:57:27,867 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:57:27,867 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:57:27,867 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:57:27,867 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:57:27,868 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:27,868 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:27,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:27,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:27,868 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:27,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:27,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:27,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:27,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:27,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:27,869 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:27,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:27,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:27,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:27,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:27,869 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:27,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:27,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,871 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,871 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:27,871 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:27,871 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:27,871 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:57:27,871 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:57:27,871 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:57:27,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:27,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:27,871 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:27,871 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:57:27,871 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,871 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:27,871 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:27,871 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:27,871 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:57:27,871 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:27,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:27,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:27,972 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:27,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:27,973 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:57:27,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:57:27,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:57:27,974 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:57:27,974 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:27,974 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:57:27,974 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:57:27,974 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:27,975 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:27,975 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,975 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:27,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:27,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:27,975 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,975 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:57:27,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:27,976 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:27,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:57:27,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:27,976 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:57:27,976 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:57:27,976 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:57:27,976 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:57:27,976 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:27,976 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:57:27,976 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:57:27,976 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:57:27,976 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. 2023-03-21 11:57:27,976 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. started... 2023-03-21 11:57:27,976 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. as already flushing 2023-03-21 11:57:27,977 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. 2023-03-21 11:57:27,977 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:57:27,977 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:57:27,977 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,977 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:57:27,977 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:57:27,977 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:27,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:27,977 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:27,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:27,978 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:27,978 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:57:27,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:27,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:27,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:27,979 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:57:27,979 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:57:27,979 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@688b4541[Count = 0] remaining members to acquire global barrier 2023-03-21 11:57:27,979 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:27,979 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:27,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:27,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:27,979 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:57:27,979 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,979 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:57:27,979 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:57:27,979 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:57:27,980 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,980 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:57:27,980 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:27,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:27,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:27,980 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:57:27,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:27,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:27,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:27,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:27,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:27,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:57:27,982 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:57:27,982 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:57:27,982 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:57:27,982 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:57:27,982 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:57:27,982 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:57:27,983 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:27,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:27,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:27,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:27,983 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:27,983 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:27,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:27,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:27,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:27,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:27,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:27,983 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:27,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:27,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:27,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:27,985 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:27,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:27,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,986 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:27,986 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,986 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:57:27,986 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:27,986 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:57:27,986 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:27,986 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:57:27,986 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:57:27,986 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:27,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:27,986 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:57:27,986 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:27,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:27,987 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:27,987 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:27,987 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:27,987 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:27,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:27,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:28,087 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:28,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:28,088 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:57:28,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:57:28,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:57:28,089 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:57:28,089 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:28,089 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:57:28,090 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:57:28,090 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:28,090 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:28,090 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:28,090 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:28,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:28,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:28,091 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:28,091 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:57:28,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:28,091 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:28,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:57:28,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:28,091 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:57:28,091 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:57:28,091 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:57:28,092 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:57:28,092 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:28,092 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:57:28,092 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:57:28,092 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:57:28,092 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. 2023-03-21 11:57:28,092 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. started... 2023-03-21 11:57:28,092 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. as already flushing 2023-03-21 11:57:28,092 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. 2023-03-21 11:57:28,092 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:57:28,092 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:57:28,092 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:28,092 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:57:28,092 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:57:28,093 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:28,093 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:28,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:28,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:28,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:28,093 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:28,093 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:57:28,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:28,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:28,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:28,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:28,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:28,094 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:57:28,094 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:57:28,094 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ff8f06b[Count = 0] remaining members to acquire global barrier 2023-03-21 11:57:28,094 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:28,095 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:28,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:28,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:28,095 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:57:28,095 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:57:28,095 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:28,095 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:57:28,095 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:57:28,096 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:57:28,096 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:28,096 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:28,096 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:57:28,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:28,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:28,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:28,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:28,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:28,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:28,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:28,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:28,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:28,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:28,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:57:28,097 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:57:28,097 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:57:28,098 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:57:28,098 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:57:28,098 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:57:28,098 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:57:28,098 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:28,098 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:28,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:28,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:28,098 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:28,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:28,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:28,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:28,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:28,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:28,099 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:28,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:28,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:28,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:28,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:28,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:28,100 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:28,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:28,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:28,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:28,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:28,101 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:28,101 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:28,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:28,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:28,101 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:28,101 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:28,101 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:57:28,101 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:57:28,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:28,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:28,101 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:57:28,101 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:28,101 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:28,101 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:28,101 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:28,101 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:28,101 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:57:28,101 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:28,101 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:57:28,123 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:28,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41056 deadline: 1679399908120, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:28,130 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:28,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41084 deadline: 1679399908122, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:28,133 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:28,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41068 deadline: 1679399908129, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:28,133 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:28,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41010 deadline: 1679399908130, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:28,140 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:28,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41034 deadline: 1679399908131, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:28,190 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=201 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/.tmp/B/8ddb9220aba846bb9de6729de420c999 2023-03-21 11:57:28,202 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:28,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:28,203 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:57:28,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:57:28,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:57:28,204 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:57:28,204 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:28,204 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:57:28,204 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:57:28,204 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:28,204 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:28,205 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:28,205 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:28,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:28,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:28,206 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:28,206 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:57:28,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:28,206 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:28,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:57:28,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:28,206 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:57:28,206 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:57:28,206 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:57:28,207 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:57:28,207 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:28,207 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:57:28,207 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:57:28,207 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:57:28,207 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. 2023-03-21 11:57:28,207 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. started... 2023-03-21 11:57:28,207 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. as already flushing 2023-03-21 11:57:28,207 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. 2023-03-21 11:57:28,207 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:57:28,207 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:57:28,207 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:28,207 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:57:28,207 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:57:28,208 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:28,208 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:28,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:28,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:28,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:28,208 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:28,208 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:57:28,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:28,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:28,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:28,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:28,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:28,209 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:57:28,209 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:57:28,209 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d74c1cc[Count = 0] remaining members to acquire global barrier 2023-03-21 11:57:28,209 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:28,210 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:28,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:28,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:28,210 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:57:28,210 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:57:28,210 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:57:28,210 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:28,210 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:57:28,211 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:28,211 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:57:28,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:28,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:28,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:28,211 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:28,211 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:57:28,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:28,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:28,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:28,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:28,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:28,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:28,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:28,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:57:28,212 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:57:28,212 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:57:28,212 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:57:28,212 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:57:28,212 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:57:28,212 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:57:28,213 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:28,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:28,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:28,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:28,213 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:28,213 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:28,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:28,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:28,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:28,214 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:28,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:28,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:28,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:28,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:28,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:28,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:28,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:28,215 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:28,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:28,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:28,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:28,216 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:28,216 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:28,216 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:28,216 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:28,216 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:28,216 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:28,216 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:28,216 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:57:28,216 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:57:28,216 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:28,216 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:28,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:28,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:28,216 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:57:28,216 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:28,216 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:57:28,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:28,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:28,317 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:28,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:28,318 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:57:28,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:57:28,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:57:28,319 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:57:28,319 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:28,319 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:57:28,319 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:57:28,320 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:28,320 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:28,320 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:28,320 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:28,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:28,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:28,320 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:28,320 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:57:28,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:28,321 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:28,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:57:28,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:28,321 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:57:28,321 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:57:28,321 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:57:28,321 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:57:28,321 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:28,321 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:57:28,321 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:57:28,322 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:57:28,322 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. 2023-03-21 11:57:28,322 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. started... 2023-03-21 11:57:28,322 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. as already flushing 2023-03-21 11:57:28,322 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. 2023-03-21 11:57:28,322 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:57:28,322 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:57:28,322 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:28,322 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:57:28,322 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:57:28,323 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:28,323 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:28,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:28,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:28,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:28,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:28,323 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:28,323 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:57:28,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:28,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:28,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:28,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:28,324 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:57:28,324 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e54c683[Count = 0] remaining members to acquire global barrier 2023-03-21 11:57:28,324 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:57:28,324 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:28,325 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:28,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:28,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:28,325 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:57:28,325 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:57:28,325 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:28,325 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:57:28,325 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:57:28,326 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:57:28,326 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:28,326 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:57:28,334 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:28,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:28,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:28,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:28,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:28,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:28,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:28,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:28,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:28,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:28,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:28,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:57:28,336 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:57:28,336 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:57:28,336 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:57:28,336 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:57:28,336 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:57:28,336 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:57:28,340 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:28,340 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:28,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:28,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:28,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:28,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:28,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:28,340 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:28,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:28,340 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:28,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:28,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:28,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:28,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:28,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:28,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:28,341 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:28,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:28,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:28,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:28,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:28,342 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:28,342 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:28,342 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:28,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:28,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:28,342 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:28,342 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:28,342 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:57:28,343 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:57:28,343 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:57:28,342 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:28,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:28,343 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:57:28,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:28,343 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:28,343 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:28,343 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:28,343 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:28,343 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:57:28,443 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:28,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:28,445 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:57:28,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:57:28,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:57:28,446 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:57:28,446 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:28,446 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:57:28,446 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:57:28,446 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:28,446 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:28,447 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:28,447 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:28,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:28,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:28,447 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:28,447 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:57:28,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:28,447 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:28,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:57:28,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:28,448 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:57:28,448 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:57:28,448 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:57:28,448 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:57:28,448 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:28,448 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:57:28,448 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:57:28,448 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:57:28,448 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. 2023-03-21 11:57:28,448 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. started... 2023-03-21 11:57:28,448 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. as already flushing 2023-03-21 11:57:28,448 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. 2023-03-21 11:57:28,448 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:57:28,448 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:57:28,448 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:28,448 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:57:28,448 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:57:28,449 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:28,449 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:28,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:28,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:28,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:28,449 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:28,449 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:57:28,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:28,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:28,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:28,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:28,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:28,450 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:57:28,450 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:57:28,450 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c472145[Count = 0] remaining members to acquire global barrier 2023-03-21 11:57:28,451 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:28,451 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:28,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:28,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:28,451 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:57:28,451 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:57:28,451 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:57:28,451 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:28,451 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:57:28,452 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:28,452 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:57:28,452 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:28,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:28,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:28,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:28,452 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:57:28,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:28,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:28,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:28,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:28,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:28,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:28,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:28,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:57:28,455 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:57:28,455 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:57:28,455 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:57:28,455 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:57:28,455 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:57:28,455 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:57:28,456 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:28,456 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:28,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:28,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:28,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:28,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:28,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:28,456 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:28,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:28,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:28,456 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:28,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:28,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:28,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:28,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:28,457 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:28,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:28,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:28,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:28,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:28,458 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:28,458 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:28,458 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:28,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:28,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:28,458 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:57:28,458 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:57:28,458 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:28,458 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:28,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:28,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:28,459 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:57:28,459 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:28,459 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:57:28,459 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:28,459 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:57:28,459 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:28,459 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:28,459 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:28,559 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:28,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:28,561 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:57:28,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:57:28,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:57:28,562 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:57:28,562 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:28,562 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:57:28,562 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:57:28,562 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:28,562 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:28,563 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:28,563 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:28,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:28,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:28,563 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:28,563 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:57:28,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:28,563 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:28,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:57:28,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:28,564 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:57:28,564 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:57:28,564 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:57:28,564 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:57:28,564 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:28,564 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:57:28,564 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:57:28,564 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:57:28,564 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. 2023-03-21 11:57:28,564 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. started... 2023-03-21 11:57:28,564 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. as already flushing 2023-03-21 11:57:28,564 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. 2023-03-21 11:57:28,564 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:57:28,564 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:57:28,564 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:28,564 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:57:28,565 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:57:28,566 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:28,566 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:28,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:28,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:28,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:28,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:28,566 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:28,566 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:57:28,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:28,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:28,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:28,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:28,567 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:57:28,567 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:57:28,567 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@14c4ef0e[Count = 0] remaining members to acquire global barrier 2023-03-21 11:57:28,567 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:28,568 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:28,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:28,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:28,568 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:57:28,568 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:57:28,568 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:57:28,568 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:28,568 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:57:28,569 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:57:28,569 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:28,569 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:28,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:28,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:28,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:28,569 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:57:28,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:28,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:28,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:28,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:28,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:28,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:28,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:28,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:57:28,570 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:57:28,571 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:57:28,571 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:57:28,571 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:57:28,571 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:57:28,571 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:57:28,571 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:28,571 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:28,571 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:28,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:28,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:28,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:28,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:28,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:28,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:28,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:28,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:28,572 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:28,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:28,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:28,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:28,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:28,573 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:28,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:28,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:28,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:28,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:28,574 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:28,574 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:28,574 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:28,574 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:28,574 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:28,574 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:57:28,574 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:28,574 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:57:28,574 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:57:28,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:28,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:28,574 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:28,574 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:28,574 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:28,574 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:57:28,574 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:28,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:28,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:28,575 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:57:28,598 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=201 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/.tmp/C/54ac64d3d21e482eb13d29075613db01 2023-03-21 11:57:28,601 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/.tmp/A/977fbb51bd5f4334a901ed7bcde18748 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/A/977fbb51bd5f4334a901ed7bcde18748 2023-03-21 11:57:28,604 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/A/977fbb51bd5f4334a901ed7bcde18748, entries=200, sequenceid=201, filesize=32.1 K 2023-03-21 11:57:28,604 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/.tmp/B/8ddb9220aba846bb9de6729de420c999 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/B/8ddb9220aba846bb9de6729de420c999 2023-03-21 11:57:28,607 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/B/8ddb9220aba846bb9de6729de420c999, entries=150, sequenceid=201, filesize=11.8 K 2023-03-21 11:57:28,607 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/.tmp/C/54ac64d3d21e482eb13d29075613db01 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/C/54ac64d3d21e482eb13d29075613db01 2023-03-21 11:57:28,610 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/C/54ac64d3d21e482eb13d29075613db01, entries=150, sequenceid=201, filesize=11.8 K 2023-03-21 11:57:28,611 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for c7c660a43107606cd80c805b0450f72f in 1654ms, sequenceid=201, compaction requested=true 2023-03-21 11:57:28,611 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c7c660a43107606cd80c805b0450f72f: 2023-03-21 11:57:28,611 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-21 11:57:28,611 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-21 11:57:28,611 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 10 store files, 0 compacting, 10 eligible, 16 blocking 2023-03-21 11:57:28,611 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 10 store files, 0 compacting, 10 eligible, 16 blocking 2023-03-21 11:57:28,611 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-21 11:57:28,613 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 10 files of size 119713 starting at candidate #0 after considering 36 permutations with 36 in ratio 2023-03-21 11:57:28,613 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 10 files of size 265303 starting at candidate #0 after considering 36 permutations with 36 in ratio 2023-03-21 11:57:28,613 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1912): c7c660a43107606cd80c805b0450f72f/B is initiating minor compaction (all files) 2023-03-21 11:57:28,613 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1912): c7c660a43107606cd80c805b0450f72f/A is initiating minor compaction (all files) 2023-03-21 11:57:28,613 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c7c660a43107606cd80c805b0450f72f/B in TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. 2023-03-21 11:57:28,613 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c7c660a43107606cd80c805b0450f72f/A in TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. 2023-03-21 11:57:28,613 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/B/74bb072119614bc8826634d4ef34469c, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/B/991eadee7bb34891bc0965cc68573ab9, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/B/1e6754799ea84d7b9bf9c471e3b74350, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/B/1a47576064c94558bba036d1819768a0, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/B/a211ea3458014176b43db706fe04b0e4, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/B/55de6e2feb69480ab42f0660aaf24831, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/B/48abbb6bd42b4d5796b7537472f0b3b0, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/B/113525b06f8742068b3bc866ee2e4eee, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/B/6f6a7ab02ea443d8a9686845d519df44, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/B/8ddb9220aba846bb9de6729de420c999] into tmpdir=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/.tmp, totalSize=116.9 K 2023-03-21 11:57:28,613 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/A/762370c7f513458aa58cc249ca0eabc1, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/A/eac2a454f4f8402482e4c0b851064b97, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/A/e260dd6de6164fdaa731e17ff288ce0e, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/A/b7247d124275490e90d3d0f3d4e5cb8b, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/A/d902305aaee54d46ad041bf60b640a52, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/A/85d8dd2b89324fcebc7c831cc96071c9, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/A/01282560f5c44eaaa13fa7ef2aa51c63, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/A/9a13801312144409a5626ba50cea90a9, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/A/bb9922a3268644ad819a55cf27aaabb0, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/A/977fbb51bd5f4334a901ed7bcde18748] into tmpdir=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/.tmp, totalSize=259.1 K 2023-03-21 11:57:28,614 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 74bb072119614bc8826634d4ef34469c, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=53, earliestPutTs=1679399835136 2023-03-21 11:57:28,614 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 762370c7f513458aa58cc249ca0eabc1, keycount=150, bloomtype=ROW, size=25.2 K, encoding=NONE, compression=NONE, seqNum=53, earliestPutTs=1679399835136 2023-03-21 11:57:28,614 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 991eadee7bb34891bc0965cc68573ab9, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=69, earliestPutTs=1679399835469 2023-03-21 11:57:28,614 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting eac2a454f4f8402482e4c0b851064b97, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=69, earliestPutTs=1679399835469 2023-03-21 11:57:28,614 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting e260dd6de6164fdaa731e17ff288ce0e, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=87, earliestPutTs=1679399836613 2023-03-21 11:57:28,614 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 1e6754799ea84d7b9bf9c471e3b74350, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=87, earliestPutTs=1679399836613 2023-03-21 11:57:28,614 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting b7247d124275490e90d3d0f3d4e5cb8b, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=103, earliestPutTs=1679399837744 2023-03-21 11:57:28,614 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 1a47576064c94558bba036d1819768a0, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=103, earliestPutTs=1679399837744 2023-03-21 11:57:28,615 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting d902305aaee54d46ad041bf60b640a52, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=120, earliestPutTs=1679399838890 2023-03-21 11:57:28,615 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting a211ea3458014176b43db706fe04b0e4, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=120, earliestPutTs=1679399838890 2023-03-21 11:57:28,615 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 85d8dd2b89324fcebc7c831cc96071c9, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=135, earliestPutTs=1679399840032 2023-03-21 11:57:28,615 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 55de6e2feb69480ab42f0660aaf24831, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=135, earliestPutTs=1679399840032 2023-03-21 11:57:28,615 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 01282560f5c44eaaa13fa7ef2aa51c63, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=152, earliestPutTs=1679399842174 2023-03-21 11:57:28,615 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 48abbb6bd42b4d5796b7537472f0b3b0, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=152, earliestPutTs=1679399842174 2023-03-21 11:57:28,615 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 9a13801312144409a5626ba50cea90a9, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=167, earliestPutTs=1679399843319 2023-03-21 11:57:28,615 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 113525b06f8742068b3bc866ee2e4eee, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=167, earliestPutTs=1679399843319 2023-03-21 11:57:28,616 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting bb9922a3268644ad819a55cf27aaabb0, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=185, earliestPutTs=1679399844485 2023-03-21 11:57:28,616 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 6f6a7ab02ea443d8a9686845d519df44, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=185, earliestPutTs=1679399844485 2023-03-21 11:57:28,616 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 977fbb51bd5f4334a901ed7bcde18748, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=201, earliestPutTs=1679399846644 2023-03-21 11:57:28,616 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 8ddb9220aba846bb9de6729de420c999, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=201, earliestPutTs=1679399846645 2023-03-21 11:57:28,634 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] throttle.PressureAwareThroughputController(145): c7c660a43107606cd80c805b0450f72f#B#compaction#321 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-21 11:57:28,637 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] throttle.PressureAwareThroughputController(145): c7c660a43107606cd80c805b0450f72f#A#compaction#322 average throughput is 3.29 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-21 11:57:28,643 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/.tmp/B/be0968d822694d14bf850e01333ba4ea as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/B/be0968d822694d14bf850e01333ba4ea 2023-03-21 11:57:28,647 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 10 (all) file(s) in c7c660a43107606cd80c805b0450f72f/B of c7c660a43107606cd80c805b0450f72f into be0968d822694d14bf850e01333ba4ea(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-21 11:57:28,647 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c7c660a43107606cd80c805b0450f72f: 2023-03-21 11:57:28,647 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f., storeName=c7c660a43107606cd80c805b0450f72f/B, priority=6, startTime=1679399848611; duration=0sec 2023-03-21 11:57:28,647 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-21 11:57:28,647 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 10 store files, 0 compacting, 10 eligible, 16 blocking 2023-03-21 11:57:28,649 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 10 files of size 119713 starting at candidate #0 after considering 36 permutations with 36 in ratio 2023-03-21 11:57:28,649 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1912): c7c660a43107606cd80c805b0450f72f/C is initiating minor compaction (all files) 2023-03-21 11:57:28,649 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c7c660a43107606cd80c805b0450f72f/C in TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. 2023-03-21 11:57:28,649 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/C/ad148fb307754ab2a9c23757fdfec04a, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/C/b16d3d1073cb49b1b291c223cec3d525, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/C/37e03c26e78c41379b2738a51d19d6ba, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/C/61d1a704c7584f7ebbfeb19cdd356840, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/C/1ee84e90c2e64473ae96854d55e2625a, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/C/0e2f9e6c7e9b4c6486bb24b96c320049, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/C/b35a675ef3f44edcac89a939f27ca34e, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/C/14a2181438834a78bbfc31c56abe9522, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/C/5dcf26970f6a4201a8d85a9190acb7e1, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/C/54ac64d3d21e482eb13d29075613db01] into tmpdir=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/.tmp, totalSize=116.9 K 2023-03-21 11:57:28,649 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting ad148fb307754ab2a9c23757fdfec04a, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=53, earliestPutTs=1679399835136 2023-03-21 11:57:28,650 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting b16d3d1073cb49b1b291c223cec3d525, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=69, earliestPutTs=1679399835469 2023-03-21 11:57:28,650 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 37e03c26e78c41379b2738a51d19d6ba, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=87, earliestPutTs=1679399836613 2023-03-21 11:57:28,650 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 61d1a704c7584f7ebbfeb19cdd356840, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=103, earliestPutTs=1679399837744 2023-03-21 11:57:28,650 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 1ee84e90c2e64473ae96854d55e2625a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=120, earliestPutTs=1679399838890 2023-03-21 11:57:28,651 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 0e2f9e6c7e9b4c6486bb24b96c320049, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=135, earliestPutTs=1679399840032 2023-03-21 11:57:28,651 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting b35a675ef3f44edcac89a939f27ca34e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=152, earliestPutTs=1679399842174 2023-03-21 11:57:28,651 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 14a2181438834a78bbfc31c56abe9522, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=167, earliestPutTs=1679399843319 2023-03-21 11:57:28,651 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 5dcf26970f6a4201a8d85a9190acb7e1, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=185, earliestPutTs=1679399844485 2023-03-21 11:57:28,651 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 54ac64d3d21e482eb13d29075613db01, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=201, earliestPutTs=1679399846645 2023-03-21 11:57:28,661 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] throttle.PressureAwareThroughputController(145): c7c660a43107606cd80c805b0450f72f#C#compaction#323 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-21 11:57:28,675 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:28,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:28,677 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:57:28,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:57:28,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:57:28,678 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:57:28,678 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:28,678 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:57:28,678 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:57:28,678 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:28,678 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:28,679 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:28,679 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:28,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:28,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:28,679 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:28,679 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:57:28,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:28,679 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:28,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:57:28,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:28,680 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:57:28,680 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:57:28,680 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:57:28,680 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:57:28,680 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:28,680 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:57:28,680 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:57:28,680 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:57:28,680 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. 2023-03-21 11:57:28,680 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. started... 2023-03-21 11:57:28,680 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c7c660a43107606cd80c805b0450f72f 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-03-21 11:57:28,681 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c7c660a43107606cd80c805b0450f72f, store=A 2023-03-21 11:57:28,681 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:57:28,681 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c7c660a43107606cd80c805b0450f72f, store=B 2023-03-21 11:57:28,681 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:57:28,681 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c7c660a43107606cd80c805b0450f72f, store=C 2023-03-21 11:57:28,681 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:57:28,698 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230321c1141e6dc24443c5b5676872532119fa to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230321c1141e6dc24443c5b5676872532119fa 2023-03-21 11:57:28,707 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=218, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/.tmp/A/6e24eaacacfe4052b0101913e612dd43 2023-03-21 11:57:29,070 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/.tmp/C/fa920748821447e09c893b323b0c879d as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/C/fa920748821447e09c893b323b0c879d 2023-03-21 11:57:29,073 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 10 (all) file(s) in c7c660a43107606cd80c805b0450f72f/C of c7c660a43107606cd80c805b0450f72f into fa920748821447e09c893b323b0c879d(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-21 11:57:29,073 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c7c660a43107606cd80c805b0450f72f: 2023-03-21 11:57:29,073 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f., storeName=c7c660a43107606cd80c805b0450f72f/C, priority=6, startTime=1679399848611; duration=0sec 2023-03-21 11:57:29,073 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-21 11:57:29,119 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=218 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/.tmp/B/1d7fdef84a2d4b0d964a67329083a097 2023-03-21 11:57:29,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(9158): Flush requested on c7c660a43107606cd80c805b0450f72f 2023-03-21 11:57:29,143 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. as already flushing 2023-03-21 11:57:29,163 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:29,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41068 deadline: 1679399909152, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:29,164 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:29,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41056 deadline: 1679399909157, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:29,164 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:29,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41034 deadline: 1679399909157, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:29,164 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:29,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41084 deadline: 1679399909158, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:29,170 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:29,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41010 deadline: 1679399909161, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:29,269 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:29,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41068 deadline: 1679399909264, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:29,270 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:29,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41056 deadline: 1679399909265, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:29,270 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:29,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41034 deadline: 1679399909265, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:29,270 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:29,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41084 deadline: 1679399909265, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:29,276 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:29,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41010 deadline: 1679399909271, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:29,451 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/.tmp/A/f754beb99d8a40328928e10d49d0dfae as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/A/f754beb99d8a40328928e10d49d0dfae 2023-03-21 11:57:29,455 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1652): Completed compaction of 10 (all) file(s) in c7c660a43107606cd80c805b0450f72f/A of c7c660a43107606cd80c805b0450f72f into f754beb99d8a40328928e10d49d0dfae(size=25.7 K), total size for store is 25.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-21 11:57:29,455 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c7c660a43107606cd80c805b0450f72f: 2023-03-21 11:57:29,455 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f., storeName=c7c660a43107606cd80c805b0450f72f/A, priority=6, startTime=1679399848611; duration=0sec 2023-03-21 11:57:29,455 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-21 11:57:29,473 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:29,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41084 deadline: 1679399909471, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:29,473 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:29,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41068 deadline: 1679399909471, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:29,474 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:29,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41056 deadline: 1679399909471, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:29,474 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:29,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41034 deadline: 1679399909471, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:29,480 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:29,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41010 deadline: 1679399909478, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:29,532 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=218 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/.tmp/C/16bb3fdc46dc40439719ee75fd72be1c 2023-03-21 11:57:29,535 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/.tmp/A/6e24eaacacfe4052b0101913e612dd43 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/A/6e24eaacacfe4052b0101913e612dd43 2023-03-21 11:57:29,538 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/A/6e24eaacacfe4052b0101913e612dd43, entries=150, sequenceid=218, filesize=25.3 K 2023-03-21 11:57:29,539 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/.tmp/B/1d7fdef84a2d4b0d964a67329083a097 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/B/1d7fdef84a2d4b0d964a67329083a097 2023-03-21 11:57:29,542 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/B/1d7fdef84a2d4b0d964a67329083a097, entries=150, sequenceid=218, filesize=11.8 K 2023-03-21 11:57:29,542 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/.tmp/C/16bb3fdc46dc40439719ee75fd72be1c as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/C/16bb3fdc46dc40439719ee75fd72be1c 2023-03-21 11:57:29,545 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/C/16bb3fdc46dc40439719ee75fd72be1c, entries=150, sequenceid=218, filesize=11.8 K 2023-03-21 11:57:29,545 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for c7c660a43107606cd80c805b0450f72f in 865ms, sequenceid=218, compaction requested=false 2023-03-21 11:57:29,545 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c7c660a43107606cd80c805b0450f72f: 2023-03-21 11:57:29,545 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. 2023-03-21 11:57:29,546 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:57:29,546 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:57:29,546 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:29,546 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:57:29,546 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:57:29,546 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:29,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:29,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:29,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:29,547 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:29,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:29,547 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:29,547 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:57:29,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:29,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:29,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:29,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:29,548 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:57:29,548 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7fd536fb[Count = 0] remaining members to acquire global barrier 2023-03-21 11:57:29,548 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:57:29,548 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:29,548 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:29,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:29,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:29,548 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:57:29,548 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:57:29,549 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:57:29,549 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:29,549 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:57:29,549 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:29,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:29,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:29,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:29,549 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:57:29,549 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:29,549 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:57:29,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:29,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:29,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:29,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:29,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:29,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:29,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:29,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:57:29,551 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:57:29,551 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:57:29,551 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:57:29,551 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:57:29,551 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:57:29,551 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:57:29,552 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:29,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:29,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:29,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:29,552 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:29,552 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:29,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:29,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:29,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:29,553 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:29,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:29,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:29,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:29,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:29,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:29,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:29,555 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:29,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:29,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:29,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:29,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:29,556 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:29,556 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:29,556 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:29,556 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:29,556 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:57:29,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:29,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:29,556 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:29,556 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:57:29,556 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:57:29,556 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:29,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:29,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:29,556 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:57:29,556 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:29,556 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:57:29,556 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:29,556 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:29,556 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:29,656 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:29,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:29,658 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:57:29,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:57:29,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:57:29,659 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:57:29,659 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:29,659 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:57:29,659 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:57:29,659 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:29,659 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:29,660 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:29,660 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:29,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:29,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:29,660 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:29,660 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:57:29,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:29,661 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:29,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:57:29,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:29,661 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:57:29,661 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:57:29,661 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:57:29,661 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:57:29,661 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:29,661 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:57:29,661 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:57:29,661 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:57:29,662 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. 2023-03-21 11:57:29,662 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. started... 2023-03-21 11:57:29,662 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c7c660a43107606cd80c805b0450f72f 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-03-21 11:57:29,662 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c7c660a43107606cd80c805b0450f72f, store=A 2023-03-21 11:57:29,662 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:57:29,662 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c7c660a43107606cd80c805b0450f72f, store=B 2023-03-21 11:57:29,662 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:57:29,662 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c7c660a43107606cd80c805b0450f72f, store=C 2023-03-21 11:57:29,662 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:57:29,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(9158): Flush requested on c7c660a43107606cd80c805b0450f72f 2023-03-21 11:57:29,777 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. as already flushing 2023-03-21 11:57:29,790 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:29,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41068 deadline: 1679399909785, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:29,791 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:29,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41010 deadline: 1679399909786, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:29,792 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:29,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41034 deadline: 1679399909787, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:29,792 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:29,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41084 deadline: 1679399909787, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:29,792 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:29,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41056 deadline: 1679399909791, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:29,896 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:29,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41068 deadline: 1679399909892, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:29,896 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:29,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41010 deadline: 1679399909892, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:29,897 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:29,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41034 deadline: 1679399909893, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:29,897 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:29,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41084 deadline: 1679399909893, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:29,897 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:29,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41056 deadline: 1679399909893, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:30,075 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303214b74216033ca4f09a26176a6d3fc61f9 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303214b74216033ca4f09a26176a6d3fc61f9 2023-03-21 11:57:30,101 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:30,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41010 deadline: 1679399910097, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:30,101 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:30,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41034 deadline: 1679399910097, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:30,101 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:30,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41056 deadline: 1679399910098, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:30,101 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:30,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41068 deadline: 1679399910098, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:30,102 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:30,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41084 deadline: 1679399910099, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:30,403 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:30,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41056 deadline: 1679399910402, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:30,403 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:30,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41010 deadline: 1679399910403, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:30,403 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:30,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41034 deadline: 1679399910403, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:30,407 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:30,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41084 deadline: 1679399910403, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:30,408 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:30,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41068 deadline: 1679399910404, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:30,480 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=236, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/.tmp/A/1b22768a625e44ea89c52f5c5f4b4fd7 2023-03-21 11:57:30,889 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=236 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/.tmp/B/0e9e94fdb21348ca85793c157130785d 2023-03-21 11:57:30,897 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=236 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/.tmp/C/a59002840906422088facd432c0b913d 2023-03-21 11:57:30,900 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/.tmp/A/1b22768a625e44ea89c52f5c5f4b4fd7 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/A/1b22768a625e44ea89c52f5c5f4b4fd7 2023-03-21 11:57:30,903 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/A/1b22768a625e44ea89c52f5c5f4b4fd7, entries=150, sequenceid=236, filesize=25.3 K 2023-03-21 11:57:30,904 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/.tmp/B/0e9e94fdb21348ca85793c157130785d as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/B/0e9e94fdb21348ca85793c157130785d 2023-03-21 11:57:30,907 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/B/0e9e94fdb21348ca85793c157130785d, entries=150, sequenceid=236, filesize=11.8 K 2023-03-21 11:57:30,907 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/.tmp/C/a59002840906422088facd432c0b913d as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/C/a59002840906422088facd432c0b913d 2023-03-21 11:57:30,909 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:30,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41056 deadline: 1679399910904, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:30,910 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:30,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41034 deadline: 1679399910905, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:30,910 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:30,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41010 deadline: 1679399910907, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:30,910 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:30,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41068 deadline: 1679399910909, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:30,910 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/C/a59002840906422088facd432c0b913d, entries=150, sequenceid=236, filesize=11.8 K 2023-03-21 11:57:30,911 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:30,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41084 deadline: 1679399910909, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:30,911 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for c7c660a43107606cd80c805b0450f72f in 1249ms, sequenceid=236, compaction requested=true 2023-03-21 11:57:30,911 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c7c660a43107606cd80c805b0450f72f: 2023-03-21 11:57:30,911 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. 2023-03-21 11:57:30,911 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:57:30,911 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:57:30,911 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:30,911 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:57:30,911 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:57:30,912 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:30,912 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:30,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:30,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:30,912 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:30,912 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:57:30,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:30,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:30,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:30,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:30,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:30,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:30,913 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:57:30,913 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74d875c[Count = 0] remaining members to acquire global barrier 2023-03-21 11:57:30,913 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:57:30,913 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:30,914 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:30,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:30,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:30,914 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:57:30,914 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:57:30,914 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:57:30,914 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:30,914 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:57:30,915 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:30,915 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:57:30,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:30,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:30,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:30,915 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:30,915 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:57:30,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:30,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:30,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:30,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:30,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:30,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:30,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:30,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:57:30,917 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:57:30,917 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:57:30,917 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:57:30,917 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:57:30,917 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:57:30,917 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:57:30,917 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:30,917 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:30,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:30,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:30,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:30,918 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:30,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:30,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:30,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:30,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:30,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:30,918 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:30,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:30,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:30,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:30,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:30,919 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:30,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:30,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:30,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:30,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:30,920 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:30,920 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:30,920 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:30,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:30,920 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:57:30,920 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:57:30,920 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:30,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:30,920 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:30,920 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:57:30,920 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:30,920 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:57:30,921 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:57:30,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:30,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:30,920 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:30,921 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:30,921 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:30,921 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:31,021 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:31,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:31,022 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:57:31,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:57:31,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:57:31,023 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:57:31,023 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:31,023 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:57:31,023 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:57:31,026 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:31,026 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:31,026 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:31,026 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:31,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:31,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:31,026 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:31,026 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:57:31,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:31,027 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:31,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:57:31,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:31,027 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:57:31,028 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:57:31,028 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:57:31,028 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:57:31,028 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:31,028 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:57:31,028 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:57:31,028 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:57:31,028 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. 2023-03-21 11:57:31,029 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. started... 2023-03-21 11:57:31,029 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c7c660a43107606cd80c805b0450f72f 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-03-21 11:57:31,029 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c7c660a43107606cd80c805b0450f72f, store=A 2023-03-21 11:57:31,029 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:57:31,029 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c7c660a43107606cd80c805b0450f72f, store=B 2023-03-21 11:57:31,029 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:57:31,029 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c7c660a43107606cd80c805b0450f72f, store=C 2023-03-21 11:57:31,029 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:57:31,040 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303216f7be920233d4baea927571de9fdd5c6 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303216f7be920233d4baea927571de9fdd5c6 2023-03-21 11:57:31,044 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=252, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/.tmp/A/67e90e58cec0421ab796d8d3e4ef7340 2023-03-21 11:57:31,452 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=252 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/.tmp/B/85eba12e54c0473b8b7747e8cbba5138 2023-03-21 11:57:31,865 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=252 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/.tmp/C/4a51e4cb072349b9a43486e397bb77fc 2023-03-21 11:57:31,868 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/.tmp/A/67e90e58cec0421ab796d8d3e4ef7340 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/A/67e90e58cec0421ab796d8d3e4ef7340 2023-03-21 11:57:31,871 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/A/67e90e58cec0421ab796d8d3e4ef7340, entries=150, sequenceid=252, filesize=25.3 K 2023-03-21 11:57:31,872 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/.tmp/B/85eba12e54c0473b8b7747e8cbba5138 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/B/85eba12e54c0473b8b7747e8cbba5138 2023-03-21 11:57:31,875 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/B/85eba12e54c0473b8b7747e8cbba5138, entries=150, sequenceid=252, filesize=11.8 K 2023-03-21 11:57:31,879 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/.tmp/C/4a51e4cb072349b9a43486e397bb77fc as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/C/4a51e4cb072349b9a43486e397bb77fc 2023-03-21 11:57:31,888 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/C/4a51e4cb072349b9a43486e397bb77fc, entries=150, sequenceid=252, filesize=11.8 K 2023-03-21 11:57:31,888 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=0 B/0 for c7c660a43107606cd80c805b0450f72f in 859ms, sequenceid=252, compaction requested=true 2023-03-21 11:57:31,888 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c7c660a43107606cd80c805b0450f72f: 2023-03-21 11:57:31,889 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. 2023-03-21 11:57:31,889 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:57:31,889 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:57:31,889 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:31,889 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:57:31,889 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:57:31,898 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:31,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:31,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:31,898 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:31,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:31,899 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:31,899 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:57:31,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:31,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:31,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:31,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:31,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:31,905 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:57:31,905 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@52d5d3f9[Count = 0] remaining members to acquire global barrier 2023-03-21 11:57:31,905 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:57:31,906 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:31,915 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:31,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:31,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:31,915 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:57:31,915 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:57:31,915 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:57:31,916 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:31,916 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:57:31,920 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:31,920 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:57:31,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:31,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:31,920 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:31,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:31,921 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:57:31,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:31,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:31,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:31,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:31,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:31,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:31,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:31,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:57:31,923 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:57:31,923 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:57:31,923 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:57:31,923 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:57:31,923 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:57:31,923 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:57:31,928 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:31,928 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:31,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:31,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:31,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:31,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:31,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:31,928 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:31,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:31,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:31,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:31,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:31,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:31,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:31,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:31,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:31,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:31,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:31,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:31,931 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:31,932 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:31,937 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:31,937 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:31,937 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:31,937 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:31,937 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:31,937 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:57:31,937 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:57:31,937 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:31,937 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:57:31,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:31,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:31,938 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:57:31,937 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:31,937 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:31,938 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:31,938 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:31,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:31,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:32,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(9158): Flush requested on c7c660a43107606cd80c805b0450f72f 2023-03-21 11:57:32,033 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c7c660a43107606cd80c805b0450f72f 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-21 11:57:32,033 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c7c660a43107606cd80c805b0450f72f, store=A 2023-03-21 11:57:32,033 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:57:32,033 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c7c660a43107606cd80c805b0450f72f, store=B 2023-03-21 11:57:32,033 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:57:32,034 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c7c660a43107606cd80c805b0450f72f, store=C 2023-03-21 11:57:32,034 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:57:32,038 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:32,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:32,041 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:57:32,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:57:32,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:57:32,063 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:57:32,063 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:32,063 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:57:32,063 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:57:32,064 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:32,064 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:32,064 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:32,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:32,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:32,065 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:32,065 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,065 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:57:32,065 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:32,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:57:32,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:32,070 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:57:32,073 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:57:32,073 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:57:32,073 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:57:32,073 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:32,073 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:57:32,073 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:57:32,073 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:57:32,073 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. 2023-03-21 11:57:32,073 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. started... 2023-03-21 11:57:32,073 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. as already flushing 2023-03-21 11:57:32,073 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. 2023-03-21 11:57:32,073 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:57:32,073 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:57:32,073 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,074 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:57:32,074 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:57:32,078 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,078 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:32,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:32,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:32,078 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:32,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:32,078 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:57:32,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:32,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:32,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:32,079 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:57:32,079 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@615afa32[Count = 0] remaining members to acquire global barrier 2023-03-21 11:57:32,079 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:57:32,079 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:32,080 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:32,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:32,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:32,080 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:57:32,080 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:57:32,080 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:57:32,080 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,080 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:57:32,081 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,081 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:57:32,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:32,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:32,081 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:32,081 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:57:32,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:32,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:32,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:32,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:32,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:32,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:57:32,082 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:57:32,082 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:57:32,082 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:57:32,082 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:57:32,082 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:57:32,082 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:57:32,084 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1679399852061, max timestamp=1679399852082Num uniques 859; 2023-03-21 11:57:32,085 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1679399852061, max timestamp=1679399852082Num uniques 859; 2023-03-21 11:57:32,085 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-03-21 11:57:32,085 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:32,085 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:32,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:32,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:32,085 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:32,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:32,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:32,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:32,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:32,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:32,085 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-03-21 11:57:32,085 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1679399852061, max timestamp=1679399852082Num uniques 859; 2023-03-21 11:57:32,086 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-03-21 11:57:32,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:32,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:32,086 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:32,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:32,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:32,087 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:32,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:32,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,090 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:32,090 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:32,090 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:32,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:32,090 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:32,090 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:32,090 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,090 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:32,090 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:32,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:32,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:32,090 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:57:32,090 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:57:32,091 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:57:32,091 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:57:32,090 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:32,090 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:32,113 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:32,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41068 deadline: 1679399912102, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,113 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:32,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41056 deadline: 1679399912103, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,113 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:32,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41010 deadline: 1679399912103, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,114 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:32,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41084 deadline: 1679399912104, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,114 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:32,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41034 deadline: 1679399912110, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,191 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:32,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:32,192 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:57:32,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:57:32,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:57:32,194 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:57:32,194 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:32,194 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:57:32,194 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:57:32,194 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:32,194 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:32,195 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,195 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:32,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:32,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:32,195 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,195 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:57:32,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:32,195 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:32,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:57:32,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:32,196 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:57:32,196 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:57:32,196 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:57:32,196 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:57:32,196 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:32,196 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:57:32,196 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:57:32,196 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:57:32,196 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. 2023-03-21 11:57:32,196 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. started... 2023-03-21 11:57:32,196 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. as already flushing 2023-03-21 11:57:32,196 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. 2023-03-21 11:57:32,196 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:57:32,197 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:57:32,197 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,197 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:57:32,197 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:57:32,197 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:32,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:32,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:32,198 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:32,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:32,198 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:32,198 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:57:32,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:32,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:32,199 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:57:32,199 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@37ee7a0c[Count = 0] remaining members to acquire global barrier 2023-03-21 11:57:32,199 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:57:32,199 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:32,200 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:32,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:32,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:32,200 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,200 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:57:32,200 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:57:32,200 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:57:32,200 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:57:32,201 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,201 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:57:32,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:32,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:32,201 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:32,201 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:57:32,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:32,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:32,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:32,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:32,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:32,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:57:32,203 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:57:32,203 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:57:32,203 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:57:32,203 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:57:32,203 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:57:32,203 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:57:32,204 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:32,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:32,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:32,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:32,204 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:32,204 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:32,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:32,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:32,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:32,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:32,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:32,204 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:32,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:32,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:32,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:32,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,205 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:32,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:32,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,206 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:32,206 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:32,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:32,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:32,206 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,207 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:32,207 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:32,207 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:32,207 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:57:32,207 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:57:32,207 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,207 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:32,207 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:57:32,207 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:32,207 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:57:32,207 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:32,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:32,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:32,218 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:32,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41068 deadline: 1679399912214, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,218 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:32,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41056 deadline: 1679399912214, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,218 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:32,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41010 deadline: 1679399912214, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,218 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:32,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41084 deadline: 1679399912214, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,218 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:32,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41034 deadline: 1679399912215, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,307 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:32,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:32,309 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:57:32,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:57:32,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:57:32,310 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:57:32,310 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:32,310 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:57:32,310 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:57:32,310 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:32,310 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:32,311 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,311 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:32,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:32,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:32,311 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,311 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:57:32,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:32,312 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:32,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:57:32,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:32,312 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:57:32,313 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:57:32,313 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:57:32,313 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:57:32,313 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:32,313 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:57:32,319 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:57:32,319 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:57:32,319 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. 2023-03-21 11:57:32,319 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. started... 2023-03-21 11:57:32,319 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. as already flushing 2023-03-21 11:57:32,319 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. 2023-03-21 11:57:32,319 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:57:32,319 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:57:32,319 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,319 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:57:32,319 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:57:32,328 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,328 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:32,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:32,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:32,328 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:32,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:32,328 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:57:32,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:32,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:32,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:32,329 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:57:32,329 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:57:32,329 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c727cca[Count = 0] remaining members to acquire global barrier 2023-03-21 11:57:32,329 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:32,332 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:32,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:32,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:32,332 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,332 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:57:32,332 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:57:32,332 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:57:32,332 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:57:32,332 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:57:32,332 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,332 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:32,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:32,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:32,333 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:57:32,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:32,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:32,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:32,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:32,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:32,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:57:32,334 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:57:32,334 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:57:32,334 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:57:32,334 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:57:32,334 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:57:32,334 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:57:32,336 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:32,336 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:32,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:32,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:32,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:32,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:32,336 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:32,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:32,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:32,336 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:32,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:32,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:32,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:32,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:32,337 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:32,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:32,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:32,339 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,339 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:32,339 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:32,339 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,339 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:32,339 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:32,339 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:32,339 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:32,339 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:57:32,339 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:57:32,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:32,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:32,339 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:57:32,339 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:32,339 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:32,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:32,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:32,339 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:57:32,340 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:57:32,423 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:32,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41068 deadline: 1679399912419, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,424 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:32,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41010 deadline: 1679399912419, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,426 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:32,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41056 deadline: 1679399912420, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,426 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:32,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41084 deadline: 1679399912420, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,427 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:32,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41034 deadline: 1679399912420, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,440 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:32,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:32,441 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:57:32,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:57:32,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:57:32,442 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:57:32,443 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:32,443 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:57:32,443 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:57:32,443 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:32,443 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:32,443 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,443 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:32,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:32,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:32,444 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,444 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:57:32,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:32,444 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:32,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:57:32,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:32,444 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:57:32,444 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:57:32,445 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:57:32,445 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:57:32,445 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:32,445 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:57:32,445 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:57:32,445 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. 2023-03-21 11:57:32,445 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:57:32,445 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. started... 2023-03-21 11:57:32,445 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. as already flushing 2023-03-21 11:57:32,445 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. 2023-03-21 11:57:32,445 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:57:32,445 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:57:32,445 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,445 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:57:32,445 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:57:32,446 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,446 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:32,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:32,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:32,446 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:32,446 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:57:32,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:32,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:32,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:32,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:32,447 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:57:32,447 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:57:32,447 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@20c426a6[Count = 0] remaining members to acquire global barrier 2023-03-21 11:57:32,447 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:32,448 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:32,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:32,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:32,448 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:57:32,448 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:57:32,448 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,448 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:57:32,448 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:57:32,449 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,449 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:57:32,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:32,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:32,449 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:32,449 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:57:32,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:32,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:32,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:32,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:32,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:32,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:57:32,451 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:57:32,451 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:57:32,451 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:57:32,451 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:57:32,451 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:57:32,451 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:57:32,452 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:32,452 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:32,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:32,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:32,452 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:32,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:32,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:32,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:32,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:32,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:32,452 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:32,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:32,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:32,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:32,453 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:32,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:32,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:32,454 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,454 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:32,454 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:32,454 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:32,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:32,455 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:32,455 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:32,455 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:57:32,455 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:57:32,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:32,455 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:57:32,455 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,455 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:32,455 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:57:32,455 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:32,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:32,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:32,455 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:57:32,455 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:32,466 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303213e42b078cdd84b04bad68f1e67b6ec11 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303213e42b078cdd84b04bad68f1e67b6ec11 2023-03-21 11:57:32,471 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=263, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/.tmp/A/7c3a5c58b20549e8a84ede03a1250ad1 2023-03-21 11:57:32,555 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:32,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:32,557 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:57:32,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:57:32,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:57:32,558 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:57:32,558 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:32,558 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:57:32,558 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:57:32,558 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:32,558 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:32,559 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:32,559 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:32,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:32,559 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,559 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:57:32,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:32,559 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:32,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:57:32,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:32,560 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:57:32,560 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:57:32,560 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:57:32,560 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:57:32,560 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:32,560 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:57:32,560 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:57:32,560 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:57:32,560 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. 2023-03-21 11:57:32,560 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. started... 2023-03-21 11:57:32,560 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. as already flushing 2023-03-21 11:57:32,560 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. 2023-03-21 11:57:32,560 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:57:32,560 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:57:32,560 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,560 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:57:32,560 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:57:32,561 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:32,561 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:32,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:32,561 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:32,561 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:57:32,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:32,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:32,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:32,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:32,562 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:57:32,562 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16f9d3e6[Count = 0] remaining members to acquire global barrier 2023-03-21 11:57:32,562 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:57:32,562 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:32,563 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:32,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:32,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:32,563 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,563 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:57:32,563 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:57:32,563 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:57:32,563 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:57:32,578 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,578 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:57:32,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:32,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:32,578 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:32,578 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:57:32,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:32,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:32,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:32,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:32,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:32,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:57:32,580 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:57:32,580 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:57:32,580 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:57:32,580 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:57:32,580 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:57:32,580 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:57:32,582 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:32,582 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:32,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:32,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:32,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:32,582 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:32,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:32,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:32,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:32,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:32,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:32,582 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:32,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:32,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:32,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:32,583 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:32,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:32,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,584 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,584 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:32,584 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:32,584 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:32,584 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:57:32,584 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:57:32,584 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,584 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:32,584 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:32,585 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:32,585 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:57:32,585 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:57:32,585 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:57:32,585 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:32,585 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:32,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:32,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:32,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:32,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:32,685 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:32,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:32,686 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:57:32,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:57:32,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:57:32,687 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:57:32,687 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:32,687 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:57:32,687 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:57:32,688 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:32,688 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:32,688 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,688 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:32,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:32,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:32,688 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,688 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:57:32,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:32,689 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:32,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:57:32,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:32,689 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:57:32,689 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:57:32,689 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:57:32,689 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:57:32,689 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:32,689 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:57:32,690 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:57:32,690 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:57:32,690 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. 2023-03-21 11:57:32,690 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. started... 2023-03-21 11:57:32,690 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. as already flushing 2023-03-21 11:57:32,690 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. 2023-03-21 11:57:32,690 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:57:32,690 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:57:32,690 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,690 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:57:32,690 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:57:32,691 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,691 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:32,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:32,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:32,691 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:32,691 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:57:32,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:32,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:32,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:32,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:32,692 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:57:32,692 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5203f52f[Count = 0] remaining members to acquire global barrier 2023-03-21 11:57:32,692 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:57:32,692 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:32,693 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:32,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:32,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:32,693 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:57:32,693 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:57:32,693 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:57:32,693 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,693 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:57:32,693 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,693 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:57:32,694 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:32,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:32,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:32,694 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:57:32,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:32,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:32,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:32,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:32,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:32,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:57:32,696 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:57:32,696 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:57:32,696 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:57:32,696 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:57:32,696 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:57:32,696 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:57:32,696 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:32,696 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:32,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:32,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:32,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:32,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:32,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:32,696 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:32,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:32,697 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:32,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:32,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:32,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:32,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:32,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:32,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,698 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:32,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:32,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,700 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,700 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:32,700 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:32,700 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,700 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:57:32,700 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:57:32,700 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:57:32,700 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:32,700 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:32,700 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:32,700 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:57:32,700 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:32,700 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:32,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:32,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:32,700 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:32,700 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:57:32,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:32,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:32,730 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:32,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41010 deadline: 1679399912724, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,730 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:32,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41068 deadline: 1679399912726, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,731 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:32,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41084 deadline: 1679399912728, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,735 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:32,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41056 deadline: 1679399912729, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,736 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:32,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41034 deadline: 1679399912729, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,801 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:32,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:32,802 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:57:32,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:57:32,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:57:32,803 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:57:32,803 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:32,803 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:57:32,803 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:57:32,803 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:32,803 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:32,804 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:32,804 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:32,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:32,804 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,804 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:57:32,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:32,804 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:32,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:57:32,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:32,805 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:57:32,805 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:57:32,805 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:57:32,805 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:57:32,805 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:32,805 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:57:32,805 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:57:32,805 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:57:32,805 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. 2023-03-21 11:57:32,805 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. started... 2023-03-21 11:57:32,805 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. as already flushing 2023-03-21 11:57:32,805 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. 2023-03-21 11:57:32,805 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:57:32,805 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:57:32,805 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,805 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:57:32,805 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:57:32,806 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,806 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:32,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:32,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:32,806 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:32,806 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:57:32,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:32,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:32,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:32,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:32,807 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:57:32,807 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d585789[Count = 0] remaining members to acquire global barrier 2023-03-21 11:57:32,807 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:57:32,807 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:32,808 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:32,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:32,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:32,808 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:57:32,808 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:57:32,808 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:57:32,808 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,808 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:57:32,809 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,809 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:57:32,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:32,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:32,809 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:32,809 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:57:32,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:32,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:32,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:32,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:32,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:32,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:57:32,810 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:57:32,810 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:57:32,810 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:57:32,810 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:57:32,810 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:57:32,810 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:57:32,811 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:32,811 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:32,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:32,811 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:32,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:32,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:32,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:32,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:32,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:32,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:32,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:32,812 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:32,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:32,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:32,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:32,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,813 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:32,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:32,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,814 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,814 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:32,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:32,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:32,814 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:32,814 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:32,814 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:32,814 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:57:32,815 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:57:32,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:32,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:32,815 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:32,815 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:57:32,815 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,815 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:57:32,815 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:32,815 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:32,815 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:32,880 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=263 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/.tmp/B/24163a51fa4d4bb89968ffb5887442ed 2023-03-21 11:57:32,915 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:32,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:32,917 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:57:32,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:57:32,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:57:32,918 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:57:32,918 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:32,918 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:57:32,918 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:57:32,918 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:32,918 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:32,919 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,919 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:32,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:32,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:32,919 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,919 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:57:32,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:32,920 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:32,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:57:32,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:32,920 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:57:32,920 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:57:32,920 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:57:32,920 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:57:32,920 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:32,920 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:57:32,920 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:57:32,921 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:57:32,921 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. 2023-03-21 11:57:32,921 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. started... 2023-03-21 11:57:32,921 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. as already flushing 2023-03-21 11:57:32,921 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. 2023-03-21 11:57:32,921 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:57:32,921 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:57:32,921 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,921 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:57:32,921 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:57:32,922 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,922 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:32,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:32,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:32,922 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:32,922 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:57:32,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:32,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:32,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:32,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:32,923 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:57:32,923 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c28b392[Count = 0] remaining members to acquire global barrier 2023-03-21 11:57:32,923 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:57:32,923 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:32,923 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:32,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:32,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:32,924 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,924 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:57:32,924 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:57:32,924 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:57:32,924 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:57:32,924 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:57:32,924 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:32,924 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:32,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:32,924 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:57:32,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:32,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:32,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:32,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:32,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:32,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:57:32,926 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:57:32,926 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:57:32,926 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:57:32,926 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:57:32,926 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:57:32,926 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:57:32,927 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:32,927 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:32,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:32,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:32,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:32,927 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:32,927 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:32,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:32,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:32,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:32,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:32,928 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:32,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:32,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:32,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:32,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,928 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:32,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:32,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,930 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,930 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:32,930 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:32,930 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:32,930 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:57:32,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:32,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:32,930 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:57:32,930 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:32,930 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:32,930 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:32,930 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:57:32,930 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:32,930 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:57:32,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:32,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:32,930 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:32,930 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:33,030 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:33,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:33,032 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:57:33,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:57:33,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:57:33,033 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:57:33,033 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:33,033 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:57:33,033 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:57:33,033 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:33,033 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:33,034 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:33,034 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:33,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:33,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:33,034 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:33,034 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:57:33,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:33,034 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:33,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:57:33,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:33,035 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:57:33,035 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:57:33,035 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:57:33,035 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:57:33,035 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:33,035 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:57:33,035 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:57:33,035 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:57:33,035 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. 2023-03-21 11:57:33,035 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. started... 2023-03-21 11:57:33,035 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. as already flushing 2023-03-21 11:57:33,035 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. 2023-03-21 11:57:33,035 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:57:33,035 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:57:33,035 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:33,035 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:57:33,035 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:57:33,036 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:33,036 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:33,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:33,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:33,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:33,036 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:33,036 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:57:33,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:33,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:33,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:33,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:33,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:33,038 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:57:33,038 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@44962512[Count = 0] remaining members to acquire global barrier 2023-03-21 11:57:33,038 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:57:33,038 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:33,039 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:33,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:33,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:33,039 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:57:33,039 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:57:33,039 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:33,039 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:57:33,039 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:57:33,039 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:57:33,039 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:33,039 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:33,039 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:57:33,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:33,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:33,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:33,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:33,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:33,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:33,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:33,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:33,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:33,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:33,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:57:33,041 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:57:33,041 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:57:33,041 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:57:33,041 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:57:33,041 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:57:33,041 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:57:33,042 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:33,042 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:33,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:33,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:33,042 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:33,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:33,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:33,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:33,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:33,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:33,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:33,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:33,043 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:33,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:33,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:33,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:33,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:33,044 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:33,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:33,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:33,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:33,047 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:33,047 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:33,047 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:33,047 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:57:33,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:33,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:33,047 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:33,047 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:57:33,047 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:57:33,047 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:33,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:33,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:33,047 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:33,047 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:33,047 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:33,047 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:33,047 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:33,047 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:57:33,048 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:57:33,148 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:33,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:33,150 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:57:33,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:57:33,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:57:33,150 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:57:33,151 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:33,151 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:57:33,151 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:57:33,151 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:33,151 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:33,152 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:33,152 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:33,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:33,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:33,152 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:33,152 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:57:33,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:33,152 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:33,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:57:33,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:33,153 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:57:33,153 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:57:33,153 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:57:33,153 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:57:33,153 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:33,153 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:57:33,153 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:57:33,153 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:57:33,153 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. 2023-03-21 11:57:33,153 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. started... 2023-03-21 11:57:33,153 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. as already flushing 2023-03-21 11:57:33,153 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. 2023-03-21 11:57:33,153 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:57:33,153 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:57:33,153 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:33,153 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:57:33,153 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:57:33,154 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:33,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:33,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:33,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:33,154 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:33,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:33,154 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:33,154 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:57:33,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:33,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:33,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:33,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:33,155 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:57:33,155 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@9f10a22[Count = 0] remaining members to acquire global barrier 2023-03-21 11:57:33,155 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:57:33,155 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:33,156 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:33,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:33,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:33,156 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:57:33,156 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:57:33,156 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:57:33,156 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:33,156 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:57:33,157 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:33,157 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:57:33,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:33,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:33,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:33,157 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:33,157 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:57:33,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:33,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:33,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:33,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:33,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:33,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:33,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:33,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:57:33,158 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:57:33,158 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:57:33,158 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:57:33,158 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:57:33,158 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:57:33,158 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:57:33,159 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:33,159 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:33,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:33,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:33,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:33,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:33,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:33,159 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:33,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:33,160 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:33,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:33,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:33,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:33,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:33,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:33,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:33,161 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:33,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:33,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:33,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:33,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:33,162 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:33,162 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:33,162 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:33,162 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:33,162 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:57:33,162 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:57:33,162 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:33,162 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:33,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:33,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:33,162 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:57:33,162 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:33,162 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:57:33,162 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:57:33,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:33,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:33,162 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:33,163 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:33,163 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:33,237 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:33,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41068 deadline: 1679399913232, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:33,239 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:33,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41010 deadline: 1679399913234, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:33,240 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:33,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41084 deadline: 1679399913234, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:33,243 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:33,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41056 deadline: 1679399913237, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:33,246 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:33,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41034 deadline: 1679399913240, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:33,263 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:33,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:33,264 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:57:33,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:57:33,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:57:33,265 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:57:33,265 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:33,266 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:57:33,266 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:57:33,266 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:33,266 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:33,266 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:33,266 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:33,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:33,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:33,267 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:33,267 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:57:33,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:33,267 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:33,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:57:33,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:33,267 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:57:33,267 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:57:33,268 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:57:33,268 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:57:33,268 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:33,268 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:57:33,268 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:57:33,268 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:57:33,268 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. 2023-03-21 11:57:33,268 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. started... 2023-03-21 11:57:33,268 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. as already flushing 2023-03-21 11:57:33,268 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. 2023-03-21 11:57:33,268 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:57:33,268 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:57:33,268 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:33,268 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:57:33,268 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:57:33,269 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:33,269 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:33,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:33,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:33,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:33,269 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:33,269 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:57:33,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:33,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:33,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:33,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:33,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:33,270 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:57:33,270 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56cae08a[Count = 0] remaining members to acquire global barrier 2023-03-21 11:57:33,270 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:57:33,270 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:33,271 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:33,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:33,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:33,271 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:57:33,271 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:57:33,271 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:57:33,271 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:33,271 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:57:33,271 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:33,271 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:57:33,272 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:33,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:33,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:33,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:33,272 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:57:33,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:33,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:33,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:33,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:33,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:33,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:33,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:33,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:57:33,273 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:57:33,273 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:57:33,273 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:57:33,273 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:57:33,273 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:57:33,273 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:57:33,274 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:33,274 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:33,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:33,274 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:33,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:33,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:33,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:33,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:33,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:33,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:33,274 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:33,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:33,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:33,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:33,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:33,276 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:33,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:33,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:33,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:33,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:33,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:33,277 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:33,277 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:33,277 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:33,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:33,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:33,277 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:33,277 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:57:33,277 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:57:33,277 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:33,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:33,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:33,277 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:33,277 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:57:33,277 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:33,277 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:33,277 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:33,277 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:33,277 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:57:33,278 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:57:33,288 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=263 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/.tmp/C/b868d29e48fc40e5b4a1201712760354 2023-03-21 11:57:33,291 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/.tmp/A/7c3a5c58b20549e8a84ede03a1250ad1 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/A/7c3a5c58b20549e8a84ede03a1250ad1 2023-03-21 11:57:33,294 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/A/7c3a5c58b20549e8a84ede03a1250ad1, entries=250, sequenceid=263, filesize=39.2 K 2023-03-21 11:57:33,294 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/.tmp/B/24163a51fa4d4bb89968ffb5887442ed as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/B/24163a51fa4d4bb89968ffb5887442ed 2023-03-21 11:57:33,297 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/B/24163a51fa4d4bb89968ffb5887442ed, entries=150, sequenceid=263, filesize=11.9 K 2023-03-21 11:57:33,297 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/.tmp/C/b868d29e48fc40e5b4a1201712760354 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/C/b868d29e48fc40e5b4a1201712760354 2023-03-21 11:57:33,300 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/C/b868d29e48fc40e5b4a1201712760354, entries=150, sequenceid=263, filesize=11.9 K 2023-03-21 11:57:33,300 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=134.18 KB/137400 for c7c660a43107606cd80c805b0450f72f in 1267ms, sequenceid=263, compaction requested=true 2023-03-21 11:57:33,300 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c7c660a43107606cd80c805b0450f72f: 2023-03-21 11:57:33,301 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-21 11:57:33,301 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-21 11:57:33,301 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-21 11:57:33,301 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-21 11:57:33,301 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-21 11:57:33,302 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 144158 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-21 11:57:33,302 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 60774 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-21 11:57:33,302 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1912): c7c660a43107606cd80c805b0450f72f/A is initiating minor compaction (all files) 2023-03-21 11:57:33,302 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1912): c7c660a43107606cd80c805b0450f72f/B is initiating minor compaction (all files) 2023-03-21 11:57:33,302 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c7c660a43107606cd80c805b0450f72f/A in TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. 2023-03-21 11:57:33,302 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c7c660a43107606cd80c805b0450f72f/B in TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. 2023-03-21 11:57:33,302 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/A/f754beb99d8a40328928e10d49d0dfae, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/A/6e24eaacacfe4052b0101913e612dd43, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/A/1b22768a625e44ea89c52f5c5f4b4fd7, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/A/67e90e58cec0421ab796d8d3e4ef7340, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/A/7c3a5c58b20549e8a84ede03a1250ad1] into tmpdir=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/.tmp, totalSize=140.8 K 2023-03-21 11:57:33,302 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/B/be0968d822694d14bf850e01333ba4ea, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/B/1d7fdef84a2d4b0d964a67329083a097, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/B/0e9e94fdb21348ca85793c157130785d, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/B/85eba12e54c0473b8b7747e8cbba5138, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/B/24163a51fa4d4bb89968ffb5887442ed] into tmpdir=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/.tmp, totalSize=59.3 K 2023-03-21 11:57:33,302 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting f754beb99d8a40328928e10d49d0dfae, keycount=150, bloomtype=ROW, size=25.7 K, encoding=NONE, compression=NONE, seqNum=201, earliestPutTs=1679399846645 2023-03-21 11:57:33,302 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting be0968d822694d14bf850e01333ba4ea, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=201, earliestPutTs=1679399846645 2023-03-21 11:57:33,304 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 6e24eaacacfe4052b0101913e612dd43, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=218, earliestPutTs=1679399846985 2023-03-21 11:57:33,304 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 1d7fdef84a2d4b0d964a67329083a097, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=218, earliestPutTs=1679399846985 2023-03-21 11:57:33,304 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 1b22768a625e44ea89c52f5c5f4b4fd7, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=236, earliestPutTs=1679399849151 2023-03-21 11:57:33,304 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 0e9e94fdb21348ca85793c157130785d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=236, earliestPutTs=1679399849151 2023-03-21 11:57:33,304 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 67e90e58cec0421ab796d8d3e4ef7340, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=252, earliestPutTs=1679399849785 2023-03-21 11:57:33,304 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 85eba12e54c0473b8b7747e8cbba5138, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=252, earliestPutTs=1679399849785 2023-03-21 11:57:33,305 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 7c3a5c58b20549e8a84ede03a1250ad1, keycount=250, bloomtype=ROW, size=39.2 K, encoding=NONE, compression=NONE, seqNum=263, earliestPutTs=1679399851976 2023-03-21 11:57:33,305 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 24163a51fa4d4bb89968ffb5887442ed, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=263, earliestPutTs=1679399852031 2023-03-21 11:57:33,312 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] throttle.PressureAwareThroughputController(145): c7c660a43107606cd80c805b0450f72f#B#compaction#336 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-21 11:57:33,317 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] throttle.PressureAwareThroughputController(145): c7c660a43107606cd80c805b0450f72f#A#compaction#337 average throughput is 3.29 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-21 11:57:33,378 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:33,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:33,379 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:57:33,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:57:33,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:57:33,380 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:57:33,380 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:33,380 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:57:33,380 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:57:33,380 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:33,380 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:33,381 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:33,381 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:33,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:33,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:33,381 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:33,381 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:57:33,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:33,382 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:33,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:57:33,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:33,382 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:57:33,382 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:57:33,382 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:57:33,382 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:57:33,382 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:33,382 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:57:33,382 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:57:33,382 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:57:33,382 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. 2023-03-21 11:57:33,382 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. started... 2023-03-21 11:57:33,383 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c7c660a43107606cd80c805b0450f72f 3/3 column families, dataSize=134.18 KB heapSize=366.23 KB 2023-03-21 11:57:33,383 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c7c660a43107606cd80c805b0450f72f, store=A 2023-03-21 11:57:33,383 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-21 11:57:33,383 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c7c660a43107606cd80c805b0450f72f, store=B 2023-03-21 11:57:33,383 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-21 11:57:33,383 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c7c660a43107606cd80c805b0450f72f, store=C 2023-03-21 11:57:33,383 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-21 11:57:33,516 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/A/70b70bb7cbf84daabcbc34626c25b38f, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/A/fead8f3c4c9340758103019f767e8c95, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/A/642fec24c0934a678f032544cbf9db7f, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/A/762370c7f513458aa58cc249ca0eabc1, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/A/eac2a454f4f8402482e4c0b851064b97, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/A/e260dd6de6164fdaa731e17ff288ce0e, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/A/b7247d124275490e90d3d0f3d4e5cb8b, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/A/d902305aaee54d46ad041bf60b640a52, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/A/85d8dd2b89324fcebc7c831cc96071c9, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/A/01282560f5c44eaaa13fa7ef2aa51c63, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/A/9a13801312144409a5626ba50cea90a9, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/A/bb9922a3268644ad819a55cf27aaabb0, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/A/977fbb51bd5f4334a901ed7bcde18748] to archive 2023-03-21 11:57:33,516 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-03-21 11:57:33,518 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/A/70b70bb7cbf84daabcbc34626c25b38f to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/A/70b70bb7cbf84daabcbc34626c25b38f 2023-03-21 11:57:33,518 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/A/fead8f3c4c9340758103019f767e8c95 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/A/fead8f3c4c9340758103019f767e8c95 2023-03-21 11:57:33,519 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/A/642fec24c0934a678f032544cbf9db7f to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/A/642fec24c0934a678f032544cbf9db7f 2023-03-21 11:57:33,520 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/A/762370c7f513458aa58cc249ca0eabc1 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/A/762370c7f513458aa58cc249ca0eabc1 2023-03-21 11:57:33,521 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/A/eac2a454f4f8402482e4c0b851064b97 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/A/eac2a454f4f8402482e4c0b851064b97 2023-03-21 11:57:33,522 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/A/e260dd6de6164fdaa731e17ff288ce0e to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/A/e260dd6de6164fdaa731e17ff288ce0e 2023-03-21 11:57:33,523 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/A/b7247d124275490e90d3d0f3d4e5cb8b to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/A/b7247d124275490e90d3d0f3d4e5cb8b 2023-03-21 11:57:33,524 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/A/d902305aaee54d46ad041bf60b640a52 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/A/d902305aaee54d46ad041bf60b640a52 2023-03-21 11:57:33,524 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/A/85d8dd2b89324fcebc7c831cc96071c9 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/A/85d8dd2b89324fcebc7c831cc96071c9 2023-03-21 11:57:33,525 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/A/01282560f5c44eaaa13fa7ef2aa51c63 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/A/01282560f5c44eaaa13fa7ef2aa51c63 2023-03-21 11:57:33,526 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/A/9a13801312144409a5626ba50cea90a9 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/A/9a13801312144409a5626ba50cea90a9 2023-03-21 11:57:33,527 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/A/bb9922a3268644ad819a55cf27aaabb0 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/A/bb9922a3268644ad819a55cf27aaabb0 2023-03-21 11:57:33,528 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/A/977fbb51bd5f4334a901ed7bcde18748 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/A/977fbb51bd5f4334a901ed7bcde18748 2023-03-21 11:57:33,550 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/B/6d7e3a7a2c0f4a19864a119375c9124b, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/B/44cbf0fe5a33402287b74db3f4be270c, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/B/74bb072119614bc8826634d4ef34469c, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/B/34732a5e76eb402fb867f57d9848d0cc, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/B/991eadee7bb34891bc0965cc68573ab9, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/B/1e6754799ea84d7b9bf9c471e3b74350, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/B/1a47576064c94558bba036d1819768a0, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/B/a211ea3458014176b43db706fe04b0e4, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/B/55de6e2feb69480ab42f0660aaf24831, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/B/48abbb6bd42b4d5796b7537472f0b3b0, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/B/113525b06f8742068b3bc866ee2e4eee, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/B/6f6a7ab02ea443d8a9686845d519df44, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/B/8ddb9220aba846bb9de6729de420c999] to archive 2023-03-21 11:57:33,551 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-03-21 11:57:33,552 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/B/6d7e3a7a2c0f4a19864a119375c9124b to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/B/6d7e3a7a2c0f4a19864a119375c9124b 2023-03-21 11:57:33,553 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/B/44cbf0fe5a33402287b74db3f4be270c to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/B/44cbf0fe5a33402287b74db3f4be270c 2023-03-21 11:57:33,554 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/B/74bb072119614bc8826634d4ef34469c to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/B/74bb072119614bc8826634d4ef34469c 2023-03-21 11:57:33,555 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/B/34732a5e76eb402fb867f57d9848d0cc to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/B/34732a5e76eb402fb867f57d9848d0cc 2023-03-21 11:57:33,556 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/B/991eadee7bb34891bc0965cc68573ab9 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/B/991eadee7bb34891bc0965cc68573ab9 2023-03-21 11:57:33,557 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/B/1e6754799ea84d7b9bf9c471e3b74350 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/B/1e6754799ea84d7b9bf9c471e3b74350 2023-03-21 11:57:33,558 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/B/1a47576064c94558bba036d1819768a0 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/B/1a47576064c94558bba036d1819768a0 2023-03-21 11:57:33,559 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/B/a211ea3458014176b43db706fe04b0e4 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/B/a211ea3458014176b43db706fe04b0e4 2023-03-21 11:57:33,560 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/B/55de6e2feb69480ab42f0660aaf24831 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/B/55de6e2feb69480ab42f0660aaf24831 2023-03-21 11:57:33,561 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/B/48abbb6bd42b4d5796b7537472f0b3b0 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/B/48abbb6bd42b4d5796b7537472f0b3b0 2023-03-21 11:57:33,563 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/B/113525b06f8742068b3bc866ee2e4eee to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/B/113525b06f8742068b3bc866ee2e4eee 2023-03-21 11:57:33,577 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/B/6f6a7ab02ea443d8a9686845d519df44 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/B/6f6a7ab02ea443d8a9686845d519df44 2023-03-21 11:57:33,578 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/B/8ddb9220aba846bb9de6729de420c999 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/B/8ddb9220aba846bb9de6729de420c999 2023-03-21 11:57:33,580 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/C/c7e173a2da7346f79762e26607b8698b, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/C/721d4b9768f242c98a6af7e44f7339a0, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/C/ad148fb307754ab2a9c23757fdfec04a, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/C/ae4e6ef53e28464ba4e7ab28b17517f1, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/C/b16d3d1073cb49b1b291c223cec3d525, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/C/37e03c26e78c41379b2738a51d19d6ba, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/C/61d1a704c7584f7ebbfeb19cdd356840, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/C/1ee84e90c2e64473ae96854d55e2625a, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/C/0e2f9e6c7e9b4c6486bb24b96c320049, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/C/b35a675ef3f44edcac89a939f27ca34e, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/C/14a2181438834a78bbfc31c56abe9522, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/C/5dcf26970f6a4201a8d85a9190acb7e1, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/C/54ac64d3d21e482eb13d29075613db01] to archive 2023-03-21 11:57:33,581 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-03-21 11:57:33,582 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/C/c7e173a2da7346f79762e26607b8698b to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/C/c7e173a2da7346f79762e26607b8698b 2023-03-21 11:57:33,583 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/C/721d4b9768f242c98a6af7e44f7339a0 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/C/721d4b9768f242c98a6af7e44f7339a0 2023-03-21 11:57:33,584 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/C/ad148fb307754ab2a9c23757fdfec04a to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/C/ad148fb307754ab2a9c23757fdfec04a 2023-03-21 11:57:33,585 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/C/ae4e6ef53e28464ba4e7ab28b17517f1 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/C/ae4e6ef53e28464ba4e7ab28b17517f1 2023-03-21 11:57:33,586 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/C/b16d3d1073cb49b1b291c223cec3d525 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/C/b16d3d1073cb49b1b291c223cec3d525 2023-03-21 11:57:33,587 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/C/37e03c26e78c41379b2738a51d19d6ba to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/C/37e03c26e78c41379b2738a51d19d6ba 2023-03-21 11:57:33,588 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/C/61d1a704c7584f7ebbfeb19cdd356840 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/C/61d1a704c7584f7ebbfeb19cdd356840 2023-03-21 11:57:33,588 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/C/1ee84e90c2e64473ae96854d55e2625a to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/C/1ee84e90c2e64473ae96854d55e2625a 2023-03-21 11:57:33,589 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/C/0e2f9e6c7e9b4c6486bb24b96c320049 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/C/0e2f9e6c7e9b4c6486bb24b96c320049 2023-03-21 11:57:33,590 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/C/b35a675ef3f44edcac89a939f27ca34e to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/C/b35a675ef3f44edcac89a939f27ca34e 2023-03-21 11:57:33,591 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/C/14a2181438834a78bbfc31c56abe9522 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/C/14a2181438834a78bbfc31c56abe9522 2023-03-21 11:57:33,592 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/C/5dcf26970f6a4201a8d85a9190acb7e1 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/C/5dcf26970f6a4201a8d85a9190acb7e1 2023-03-21 11:57:33,593 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/C/54ac64d3d21e482eb13d29075613db01 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/C/54ac64d3d21e482eb13d29075613db01 2023-03-21 11:57:33,721 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/.tmp/B/f9bb6ef517cc4266a4ac9cb7a51c4446 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/B/f9bb6ef517cc4266a4ac9cb7a51c4446 2023-03-21 11:57:33,727 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in c7c660a43107606cd80c805b0450f72f/B of c7c660a43107606cd80c805b0450f72f into f9bb6ef517cc4266a4ac9cb7a51c4446(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-21 11:57:33,727 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c7c660a43107606cd80c805b0450f72f: 2023-03-21 11:57:33,727 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f., storeName=c7c660a43107606cd80c805b0450f72f/B, priority=11, startTime=1679399853301; duration=0sec 2023-03-21 11:57:33,727 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-21 11:57:33,727 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-21 11:57:33,728 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 60774 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-21 11:57:33,728 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1912): c7c660a43107606cd80c805b0450f72f/C is initiating minor compaction (all files) 2023-03-21 11:57:33,728 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c7c660a43107606cd80c805b0450f72f/C in TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. 2023-03-21 11:57:33,728 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/C/fa920748821447e09c893b323b0c879d, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/C/16bb3fdc46dc40439719ee75fd72be1c, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/C/a59002840906422088facd432c0b913d, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/C/4a51e4cb072349b9a43486e397bb77fc, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/C/b868d29e48fc40e5b4a1201712760354] into tmpdir=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/.tmp, totalSize=59.3 K 2023-03-21 11:57:33,728 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting fa920748821447e09c893b323b0c879d, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=201, earliestPutTs=1679399846645 2023-03-21 11:57:33,729 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 16bb3fdc46dc40439719ee75fd72be1c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=218, earliestPutTs=1679399846985 2023-03-21 11:57:33,729 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting a59002840906422088facd432c0b913d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=236, earliestPutTs=1679399849151 2023-03-21 11:57:33,729 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 4a51e4cb072349b9a43486e397bb77fc, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=252, earliestPutTs=1679399849785 2023-03-21 11:57:33,729 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting b868d29e48fc40e5b4a1201712760354, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=263, earliestPutTs=1679399852031 2023-03-21 11:57:33,735 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] throttle.PressureAwareThroughputController(145): c7c660a43107606cd80c805b0450f72f#C#compaction#339 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 57.69 MB/second 2023-03-21 11:57:33,743 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/.tmp/C/08bcfd2625d3482ead3195e429376f76 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/C/08bcfd2625d3482ead3195e429376f76 2023-03-21 11:57:33,746 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in c7c660a43107606cd80c805b0450f72f/C of c7c660a43107606cd80c805b0450f72f into 08bcfd2625d3482ead3195e429376f76(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-21 11:57:33,746 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c7c660a43107606cd80c805b0450f72f: 2023-03-21 11:57:33,746 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f., storeName=c7c660a43107606cd80c805b0450f72f/C, priority=11, startTime=1679399853301; duration=0sec 2023-03-21 11:57:33,746 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-21 11:57:33,793 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230321b67e02f3251740698b228bc5b9afbba9 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230321b67e02f3251740698b228bc5b9afbba9 2023-03-21 11:57:33,948 DEBUG [Thread-2239] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x52829d32 to 127.0.0.1:61761 2023-03-21 11:57:33,948 DEBUG [Thread-2239] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-21 11:57:33,950 DEBUG [Thread-2233] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1aec6811 to 127.0.0.1:61761 2023-03-21 11:57:33,950 DEBUG [Thread-2233] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-21 11:57:33,950 DEBUG [Thread-2237] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x17385c96 to 127.0.0.1:61761 2023-03-21 11:57:33,950 DEBUG [Thread-2237] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-21 11:57:33,953 DEBUG [Thread-2231] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4e90d228 to 127.0.0.1:61761 2023-03-21 11:57:33,953 DEBUG [Thread-2231] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-21 11:57:33,955 DEBUG [Thread-2235] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6d951686 to 127.0.0.1:61761 2023-03-21 11:57:33,955 DEBUG [Thread-2235] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-21 11:57:34,141 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/.tmp/A/da53bfa6bf844ad6b44847068b3aa359 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/A/da53bfa6bf844ad6b44847068b3aa359 2023-03-21 11:57:34,148 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in c7c660a43107606cd80c805b0450f72f/A of c7c660a43107606cd80c805b0450f72f into da53bfa6bf844ad6b44847068b3aa359(size=25.6 K), total size for store is 25.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-21 11:57:34,148 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c7c660a43107606cd80c805b0450f72f: 2023-03-21 11:57:34,148 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f., storeName=c7c660a43107606cd80c805b0450f72f/A, priority=11, startTime=1679399853300; duration=0sec 2023-03-21 11:57:34,148 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-21 11:57:34,205 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=296, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/.tmp/A/d41117140c3f424f8a3b5c764ea23d09 2023-03-21 11:57:34,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(9158): Flush requested on c7c660a43107606cd80c805b0450f72f 2023-03-21 11:57:34,243 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. as already flushing 2023-03-21 11:57:34,243 DEBUG [Thread-2226] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4f54fc1a to 127.0.0.1:61761 2023-03-21 11:57:34,244 DEBUG [Thread-2226] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-21 11:57:34,247 DEBUG [Thread-2228] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5c70a497 to 127.0.0.1:61761 2023-03-21 11:57:34,247 DEBUG [Thread-2228] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-21 11:57:34,248 DEBUG [Thread-2224] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x727ea8af to 127.0.0.1:61761 2023-03-21 11:57:34,248 DEBUG [Thread-2224] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-21 11:57:34,250 DEBUG [Thread-2220] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3acdb9c4 to 127.0.0.1:61761 2023-03-21 11:57:34,250 DEBUG [Thread-2220] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-21 11:57:34,254 DEBUG [Thread-2222] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3d941bc8 to 127.0.0.1:61761 2023-03-21 11:57:34,254 DEBUG [Thread-2222] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-21 11:57:34,622 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=296 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/.tmp/B/7bfa8fc824274325bfdc91bb9aec39f4 2023-03-21 11:57:35,040 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=296 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/.tmp/C/85c2c96b4d4b4081b279e9fc53fe63f5 2023-03-21 11:57:35,051 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/.tmp/A/d41117140c3f424f8a3b5c764ea23d09 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/A/d41117140c3f424f8a3b5c764ea23d09 2023-03-21 11:57:35,055 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/A/d41117140c3f424f8a3b5c764ea23d09, entries=150, sequenceid=296, filesize=25.4 K 2023-03-21 11:57:35,056 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/.tmp/B/7bfa8fc824274325bfdc91bb9aec39f4 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/B/7bfa8fc824274325bfdc91bb9aec39f4 2023-03-21 11:57:35,061 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/B/7bfa8fc824274325bfdc91bb9aec39f4, entries=150, sequenceid=296, filesize=11.9 K 2023-03-21 11:57:35,062 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/.tmp/C/85c2c96b4d4b4081b279e9fc53fe63f5 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/C/85c2c96b4d4b4081b279e9fc53fe63f5 2023-03-21 11:57:35,068 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/C/85c2c96b4d4b4081b279e9fc53fe63f5, entries=150, sequenceid=296, filesize=11.9 K 2023-03-21 11:57:35,068 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~366.19 KB/374976, currentSize=33.54 KB/34350 for c7c660a43107606cd80c805b0450f72f in 1686ms, sequenceid=296, compaction requested=false 2023-03-21 11:57:35,069 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c7c660a43107606cd80c805b0450f72f: 2023-03-21 11:57:35,069 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. 2023-03-21 11:57:35,069 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:57:35,069 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:57:35,069 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:35,069 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:57:35,069 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:57:35,070 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:35,070 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:35,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:35,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:35,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:35,071 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:35,071 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:57:35,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:35,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:35,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:35,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:35,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:35,072 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:57:35,072 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@fa77de2[Count = 0] remaining members to acquire global barrier 2023-03-21 11:57:35,072 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:57:35,072 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:35,073 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:35,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:35,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:35,073 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:35,073 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:57:35,073 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:57:35,073 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:57:35,073 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:57:35,074 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:57:35,074 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:35,074 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:35,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:35,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:35,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:35,074 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:57:35,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:35,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:35,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:35,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:35,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:35,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:35,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:35,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:57:35,076 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:57:35,076 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:57:35,076 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:57:35,076 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:57:35,076 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:57:35,076 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:57:35,076 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:35,076 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:35,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:35,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:35,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:35,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:35,076 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:35,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:35,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:35,077 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:35,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:35,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:35,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:35,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:35,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:35,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:35,077 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:35,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:35,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:35,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:35,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:35,079 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:35,079 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:35,079 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:35,079 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:57:35,079 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:57:35,079 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:35,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:35,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:35,079 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:35,079 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:35,080 DEBUG [Thread-2230] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:57:35,080 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:35,080 DEBUG [Thread-2230] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:57:35,080 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:35,080 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:35,080 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:35,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:35,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:35,180 DEBUG [Thread-2230] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:35,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:35,181 INFO [Listener at localhost.localdomain/37475] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-03-21 11:57:35,181 INFO [Listener at localhost.localdomain/37475] hbase.AcidGuaranteesTestTool(394): wrote 45 2023-03-21 11:57:35,181 INFO [Listener at localhost.localdomain/37475] hbase.AcidGuaranteesTestTool(394): wrote 52 2023-03-21 11:57:35,181 INFO [Listener at localhost.localdomain/37475] hbase.AcidGuaranteesTestTool(394): wrote 46 2023-03-21 11:57:35,181 INFO [Listener at localhost.localdomain/37475] hbase.AcidGuaranteesTestTool(394): wrote 45 2023-03-21 11:57:35,181 INFO [Listener at localhost.localdomain/37475] hbase.AcidGuaranteesTestTool(394): wrote 52 2023-03-21 11:57:35,181 INFO [Listener at localhost.localdomain/37475] hbase.AcidGuaranteesTestTool(396): Readers: 2023-03-21 11:57:35,181 INFO [Listener at localhost.localdomain/37475] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-03-21 11:57:35,181 INFO [Listener at localhost.localdomain/37475] hbase.AcidGuaranteesTestTool(402): scanned 1938 2023-03-21 11:57:35,181 INFO [Listener at localhost.localdomain/37475] hbase.AcidGuaranteesTestTool(403): verified 5811 rows 2023-03-21 11:57:35,181 INFO [Listener at localhost.localdomain/37475] hbase.AcidGuaranteesTestTool(402): scanned 1946 2023-03-21 11:57:35,181 INFO [Listener at localhost.localdomain/37475] hbase.AcidGuaranteesTestTool(403): verified 5820 rows 2023-03-21 11:57:35,181 INFO [Listener at localhost.localdomain/37475] hbase.AcidGuaranteesTestTool(402): scanned 1923 2023-03-21 11:57:35,181 INFO [Listener at localhost.localdomain/37475] hbase.AcidGuaranteesTestTool(403): verified 5769 rows 2023-03-21 11:57:35,181 INFO [Listener at localhost.localdomain/37475] hbase.AcidGuaranteesTestTool(402): scanned 1937 2023-03-21 11:57:35,181 INFO [Listener at localhost.localdomain/37475] hbase.AcidGuaranteesTestTool(403): verified 5811 rows 2023-03-21 11:57:35,181 INFO [Listener at localhost.localdomain/37475] hbase.AcidGuaranteesTestTool(402): scanned 1934 2023-03-21 11:57:35,182 INFO [Listener at localhost.localdomain/37475] hbase.AcidGuaranteesTestTool(403): verified 5802 rows 2023-03-21 11:57:35,182 INFO [Listener at localhost.localdomain/37475] client.ConnectionImplementation(1959): Closing master protocol: MasterService 2023-03-21 11:57:35,182 DEBUG [Listener at localhost.localdomain/37475] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x648e005e to 127.0.0.1:61761 2023-03-21 11:57:35,182 DEBUG [Listener at localhost.localdomain/37475] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-21 11:57:35,183 INFO [Listener at localhost.localdomain/37475] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-03-21 11:57:35,183 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.HMaster$11(2418): Client=jenkins//148.251.75.209 disable TestAcidGuarantees 2023-03-21 11:57:35,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure2.ProcedureExecutor(1029): Stored pid=43, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-03-21 11:57:35,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-03-21 11:57:35,187 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679399855187"}]},"ts":"1679399855187"} 2023-03-21 11:57:35,187 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-03-21 11:57:35,203 INFO [PEWorker-3] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-03-21 11:57:35,204 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=44, ppid=43, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=c7c660a43107606cd80c805b0450f72f, UNASSIGN}] 2023-03-21 11:57:35,204 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, ppid=43, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=c7c660a43107606cd80c805b0450f72f, UNASSIGN 2023-03-21 11:57:35,205 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=44 updating hbase:meta row=c7c660a43107606cd80c805b0450f72f, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:35,205 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679399855205"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1679399855205"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679399855205"}]},"ts":"1679399855205"} 2023-03-21 11:57:35,207 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=45, ppid=44, state=RUNNABLE; CloseRegionProcedure c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030}] 2023-03-21 11:57:35,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-03-21 11:57:35,359 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close c7c660a43107606cd80c805b0450f72f 2023-03-21 11:57:35,360 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing c7c660a43107606cd80c805b0450f72f, disabling compactions & flushes 2023-03-21 11:57:35,360 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. 2023-03-21 11:57:35,360 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. 2023-03-21 11:57:35,360 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. after waiting 0 ms 2023-03-21 11:57:35,360 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. 2023-03-21 11:57:35,360 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing c7c660a43107606cd80c805b0450f72f 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-03-21 11:57:35,361 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c7c660a43107606cd80c805b0450f72f, store=A 2023-03-21 11:57:35,361 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:57:35,361 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c7c660a43107606cd80c805b0450f72f, store=B 2023-03-21 11:57:35,361 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:57:35,361 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c7c660a43107606cd80c805b0450f72f, store=C 2023-03-21 11:57:35,361 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:57:35,379 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230321ef638480a7bf474db055acbde7e43b68 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230321ef638480a7bf474db055acbde7e43b68 2023-03-21 11:57:35,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-03-21 11:57:35,785 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=307, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/.tmp/A/10665a2fcfb640638fa028a10484c5df 2023-03-21 11:57:35,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-03-21 11:57:36,204 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=307 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/.tmp/B/719c29018ac5469f974bb5fae115b1d5 2023-03-21 11:57:36,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-03-21 11:57:36,623 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=307 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/.tmp/C/28e877b0c9e34c4c95de32896b20d012 2023-03-21 11:57:36,633 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/.tmp/A/10665a2fcfb640638fa028a10484c5df as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/A/10665a2fcfb640638fa028a10484c5df 2023-03-21 11:57:36,639 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/A/10665a2fcfb640638fa028a10484c5df, entries=150, sequenceid=307, filesize=25.4 K 2023-03-21 11:57:36,640 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/.tmp/B/719c29018ac5469f974bb5fae115b1d5 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/B/719c29018ac5469f974bb5fae115b1d5 2023-03-21 11:57:36,645 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/B/719c29018ac5469f974bb5fae115b1d5, entries=150, sequenceid=307, filesize=11.9 K 2023-03-21 11:57:36,646 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/.tmp/C/28e877b0c9e34c4c95de32896b20d012 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/C/28e877b0c9e34c4c95de32896b20d012 2023-03-21 11:57:36,651 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/C/28e877b0c9e34c4c95de32896b20d012, entries=150, sequenceid=307, filesize=11.9 K 2023-03-21 11:57:36,652 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=0 B/0 for c7c660a43107606cd80c805b0450f72f in 1292ms, sequenceid=307, compaction requested=true 2023-03-21 11:57:36,652 DEBUG [StoreCloser-TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/A/f754beb99d8a40328928e10d49d0dfae, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/A/6e24eaacacfe4052b0101913e612dd43, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/A/1b22768a625e44ea89c52f5c5f4b4fd7, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/A/67e90e58cec0421ab796d8d3e4ef7340, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/A/7c3a5c58b20549e8a84ede03a1250ad1] to archive 2023-03-21 11:57:36,653 DEBUG [StoreCloser-TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-21 11:57:36,655 DEBUG [StoreCloser-TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/A/f754beb99d8a40328928e10d49d0dfae to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/A/f754beb99d8a40328928e10d49d0dfae 2023-03-21 11:57:36,657 DEBUG [StoreCloser-TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/A/6e24eaacacfe4052b0101913e612dd43 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/A/6e24eaacacfe4052b0101913e612dd43 2023-03-21 11:57:36,658 DEBUG [StoreCloser-TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/A/1b22768a625e44ea89c52f5c5f4b4fd7 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/A/1b22768a625e44ea89c52f5c5f4b4fd7 2023-03-21 11:57:36,660 DEBUG [StoreCloser-TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/A/67e90e58cec0421ab796d8d3e4ef7340 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/A/67e90e58cec0421ab796d8d3e4ef7340 2023-03-21 11:57:36,661 DEBUG [StoreCloser-TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/A/7c3a5c58b20549e8a84ede03a1250ad1 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/A/7c3a5c58b20549e8a84ede03a1250ad1 2023-03-21 11:57:36,662 DEBUG [StoreCloser-TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/B/be0968d822694d14bf850e01333ba4ea, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/B/1d7fdef84a2d4b0d964a67329083a097, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/B/0e9e94fdb21348ca85793c157130785d, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/B/85eba12e54c0473b8b7747e8cbba5138, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/B/24163a51fa4d4bb89968ffb5887442ed] to archive 2023-03-21 11:57:36,663 DEBUG [StoreCloser-TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-21 11:57:36,665 DEBUG [StoreCloser-TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/B/be0968d822694d14bf850e01333ba4ea to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/B/be0968d822694d14bf850e01333ba4ea 2023-03-21 11:57:36,666 DEBUG [StoreCloser-TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/B/1d7fdef84a2d4b0d964a67329083a097 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/B/1d7fdef84a2d4b0d964a67329083a097 2023-03-21 11:57:36,668 DEBUG [StoreCloser-TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/B/0e9e94fdb21348ca85793c157130785d to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/B/0e9e94fdb21348ca85793c157130785d 2023-03-21 11:57:36,669 DEBUG [StoreCloser-TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/B/85eba12e54c0473b8b7747e8cbba5138 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/B/85eba12e54c0473b8b7747e8cbba5138 2023-03-21 11:57:36,671 DEBUG [StoreCloser-TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/B/24163a51fa4d4bb89968ffb5887442ed to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/B/24163a51fa4d4bb89968ffb5887442ed 2023-03-21 11:57:36,672 DEBUG [StoreCloser-TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/C/fa920748821447e09c893b323b0c879d, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/C/16bb3fdc46dc40439719ee75fd72be1c, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/C/a59002840906422088facd432c0b913d, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/C/4a51e4cb072349b9a43486e397bb77fc, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/C/b868d29e48fc40e5b4a1201712760354] to archive 2023-03-21 11:57:36,673 DEBUG [StoreCloser-TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-21 11:57:36,675 DEBUG [StoreCloser-TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/C/fa920748821447e09c893b323b0c879d to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/C/fa920748821447e09c893b323b0c879d 2023-03-21 11:57:36,676 DEBUG [StoreCloser-TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/C/16bb3fdc46dc40439719ee75fd72be1c to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/C/16bb3fdc46dc40439719ee75fd72be1c 2023-03-21 11:57:36,678 DEBUG [StoreCloser-TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/C/a59002840906422088facd432c0b913d to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/C/a59002840906422088facd432c0b913d 2023-03-21 11:57:36,679 DEBUG [StoreCloser-TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/C/4a51e4cb072349b9a43486e397bb77fc to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/C/4a51e4cb072349b9a43486e397bb77fc 2023-03-21 11:57:36,681 DEBUG [StoreCloser-TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/C/b868d29e48fc40e5b4a1201712760354 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/C/b868d29e48fc40e5b4a1201712760354 2023-03-21 11:57:36,686 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/recovered.edits/310.seqid, newMaxSeqId=310, maxSeqId=4 2023-03-21 11:57:36,688 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f. 2023-03-21 11:57:36,688 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for c7c660a43107606cd80c805b0450f72f: 2023-03-21 11:57:36,691 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed c7c660a43107606cd80c805b0450f72f 2023-03-21 11:57:36,692 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=44 updating hbase:meta row=c7c660a43107606cd80c805b0450f72f, regionState=CLOSED 2023-03-21 11:57:36,692 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679399856692"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679399856692"}]},"ts":"1679399856692"} 2023-03-21 11:57:36,695 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=45, resume processing ppid=44 2023-03-21 11:57:36,695 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=45, ppid=44, state=SUCCESS; CloseRegionProcedure c7c660a43107606cd80c805b0450f72f, server=jenkins-hbase20.apache.org,34369,1679399732030 in 1.4870 sec 2023-03-21 11:57:36,696 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=44, resume processing ppid=43 2023-03-21 11:57:36,696 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=44, ppid=43, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=c7c660a43107606cd80c805b0450f72f, UNASSIGN in 1.4910 sec 2023-03-21 11:57:36,697 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679399856697"}]},"ts":"1679399856697"} 2023-03-21 11:57:36,698 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-03-21 11:57:36,699 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-03-21 11:57:36,700 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=43, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.5160 sec 2023-03-21 11:57:37,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-03-21 11:57:37,296 INFO [Listener at localhost.localdomain/37475] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 43 completed 2023-03-21 11:57:37,298 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.HMaster$5(2228): Client=jenkins//148.251.75.209 delete TestAcidGuarantees 2023-03-21 11:57:37,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure2.ProcedureExecutor(1029): Stored pid=46, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-03-21 11:57:37,303 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=46, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-21 11:57:37,305 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=46, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-21 11:57:37,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1227): Checking to see if procedure is done pid=46 2023-03-21 11:57:37,310 DEBUG [HFileArchiver-9] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/.tmp/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f 2023-03-21 11:57:37,312 DEBUG [HFileArchiver-9] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/.tmp/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/A, FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/.tmp/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/B, FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/.tmp/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/C, FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/.tmp/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/recovered.edits] 2023-03-21 11:57:37,316 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/.tmp/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/A/10665a2fcfb640638fa028a10484c5df to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/A/10665a2fcfb640638fa028a10484c5df 2023-03-21 11:57:37,318 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/.tmp/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/A/d41117140c3f424f8a3b5c764ea23d09 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/A/d41117140c3f424f8a3b5c764ea23d09 2023-03-21 11:57:37,319 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/.tmp/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/A/da53bfa6bf844ad6b44847068b3aa359 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/A/da53bfa6bf844ad6b44847068b3aa359 2023-03-21 11:57:37,322 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/.tmp/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/B/719c29018ac5469f974bb5fae115b1d5 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/B/719c29018ac5469f974bb5fae115b1d5 2023-03-21 11:57:37,323 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/.tmp/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/B/7bfa8fc824274325bfdc91bb9aec39f4 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/B/7bfa8fc824274325bfdc91bb9aec39f4 2023-03-21 11:57:37,324 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/.tmp/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/B/f9bb6ef517cc4266a4ac9cb7a51c4446 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/B/f9bb6ef517cc4266a4ac9cb7a51c4446 2023-03-21 11:57:37,326 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/.tmp/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/C/08bcfd2625d3482ead3195e429376f76 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/C/08bcfd2625d3482ead3195e429376f76 2023-03-21 11:57:37,328 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/.tmp/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/C/28e877b0c9e34c4c95de32896b20d012 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/C/28e877b0c9e34c4c95de32896b20d012 2023-03-21 11:57:37,334 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/.tmp/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/C/85c2c96b4d4b4081b279e9fc53fe63f5 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/C/85c2c96b4d4b4081b279e9fc53fe63f5 2023-03-21 11:57:37,337 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/.tmp/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/recovered.edits/310.seqid to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f/recovered.edits/310.seqid 2023-03-21 11:57:37,337 DEBUG [HFileArchiver-9] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/.tmp/data/default/TestAcidGuarantees/c7c660a43107606cd80c805b0450f72f 2023-03-21 11:57:37,337 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-21 11:57:37,337 DEBUG [PEWorker-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-03-21 11:57:37,338 DEBUG [PEWorker-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-03-21 11:57:37,341 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032103ac55eb5ddb476d962a75690363d242 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032103ac55eb5ddb476d962a75690363d242 2023-03-21 11:57:37,342 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303210d3f9931a8e144fba951e07d7f73edf0 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303210d3f9931a8e144fba951e07d7f73edf0 2023-03-21 11:57:37,343 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032119f5396649064f43a8a71f50d58a9349 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032119f5396649064f43a8a71f50d58a9349 2023-03-21 11:57:37,344 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230321201215bd17a14f3c93087705bd32bd97 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230321201215bd17a14f3c93087705bd32bd97 2023-03-21 11:57:37,345 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032131cecce316084f0e8f4b215d8d017733 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032131cecce316084f0e8f4b215d8d017733 2023-03-21 11:57:37,346 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230321350913b8a56c4b87bbb7f45b69c730c8 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230321350913b8a56c4b87bbb7f45b69c730c8 2023-03-21 11:57:37,346 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303213e42b078cdd84b04bad68f1e67b6ec11 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303213e42b078cdd84b04bad68f1e67b6ec11 2023-03-21 11:57:37,347 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303214b74216033ca4f09a26176a6d3fc61f9 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303214b74216033ca4f09a26176a6d3fc61f9 2023-03-21 11:57:37,348 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303214efa2554ab534decaa200f37fe54a016 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303214efa2554ab534decaa200f37fe54a016 2023-03-21 11:57:37,349 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230321517e7a1441ce46e6b598c15c88621bd3 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230321517e7a1441ce46e6b598c15c88621bd3 2023-03-21 11:57:37,350 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303216f7be920233d4baea927571de9fdd5c6 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303216f7be920233d4baea927571de9fdd5c6 2023-03-21 11:57:37,351 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032179cf71e5d33749baa5cc182e1644a8df to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032179cf71e5d33749baa5cc182e1644a8df 2023-03-21 11:57:37,352 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032197af70b8569848d99a72b9b5cfedd0bc to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032197af70b8569848d99a72b9b5cfedd0bc 2023-03-21 11:57:37,353 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230321b67e02f3251740698b228bc5b9afbba9 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230321b67e02f3251740698b228bc5b9afbba9 2023-03-21 11:57:37,353 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230321c1141e6dc24443c5b5676872532119fa to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230321c1141e6dc24443c5b5676872532119fa 2023-03-21 11:57:37,354 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230321ed03bea8e6b84960a675bb9b505937f2 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230321ed03bea8e6b84960a675bb9b505937f2 2023-03-21 11:57:37,355 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230321ef638480a7bf474db055acbde7e43b68 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230321ef638480a7bf474db055acbde7e43b68 2023-03-21 11:57:37,356 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230321fd673f7ecc1f46d7a0f38f3aa4977880 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230321fd673f7ecc1f46d7a0f38f3aa4977880 2023-03-21 11:57:37,356 DEBUG [PEWorker-5] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-03-21 11:57:37,357 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=46, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-21 11:57:37,358 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-03-21 11:57:37,359 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-03-21 11:57:37,360 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=46, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-21 11:57:37,360 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-03-21 11:57:37,360 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1679399857360"}]},"ts":"9223372036854775807"} 2023-03-21 11:57:37,361 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-03-21 11:57:37,362 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => c7c660a43107606cd80c805b0450f72f, NAME => 'TestAcidGuarantees,,1679399830798.c7c660a43107606cd80c805b0450f72f.', STARTKEY => '', ENDKEY => ''}] 2023-03-21 11:57:37,362 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-03-21 11:57:37,362 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1679399857362"}]},"ts":"9223372036854775807"} 2023-03-21 11:57:37,363 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-03-21 11:57:37,364 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=46, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-21 11:57:37,364 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=46, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 66 msec 2023-03-21 11:57:37,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1227): Checking to see if procedure is done pid=46 2023-03-21 11:57:37,407 INFO [Listener at localhost.localdomain/37475] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 46 completed 2023-03-21 11:57:37,421 INFO [Listener at localhost.localdomain/37475] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobScanAtomicity Thread=232 (was 224) - Thread LEAK? -, OpenFileDescriptor=538 (was 556), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=380 (was 315) - SystemLoadAverage LEAK? -, ProcessCount=167 (was 167), AvailableMemoryMB=9089 (was 8971) - AvailableMemoryMB LEAK? - 2023-03-21 11:57:37,432 INFO [Listener at localhost.localdomain/37475] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testScanAtomicity Thread=232, OpenFileDescriptor=538, MaxFileDescriptor=60000, SystemLoadAverage=380, ProcessCount=167, AvailableMemoryMB=9089 2023-03-21 11:57:37,434 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-21 11:57:37,434 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.HMaster$4(2112): Client=jenkins//148.251.75.209 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-21 11:57:37,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure2.ProcedureExecutor(1029): Stored pid=47, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-03-21 11:57:37,436 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-03-21 11:57:37,437 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(697): Client=jenkins//148.251.75.209 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 47 2023-03-21 11:57:37,437 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-03-21 11:57:37,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-03-21 11:57:37,438 DEBUG [HFileArchiver-10] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/.tmp/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad 2023-03-21 11:57:37,438 DEBUG [HFileArchiver-10] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/.tmp/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad empty. 2023-03-21 11:57:37,439 DEBUG [HFileArchiver-10] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/.tmp/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad 2023-03-21 11:57:37,439 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-21 11:57:37,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-03-21 11:57:37,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-03-21 11:57:37,846 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-21 11:57:37,847 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 8dbd4df2f9769e1bcf08d64a8aceb9ad, NAME => 'TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/.tmp 2023-03-21 11:57:38,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-03-21 11:57:38,262 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-21 11:57:38,263 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 8dbd4df2f9769e1bcf08d64a8aceb9ad, disabling compactions & flushes 2023-03-21 11:57:38,263 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:38,263 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:38,263 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. after waiting 0 ms 2023-03-21 11:57:38,263 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:38,263 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:38,263 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 8dbd4df2f9769e1bcf08d64a8aceb9ad: 2023-03-21 11:57:38,268 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-03-21 11:57:38,270 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679399858270"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679399858270"}]},"ts":"1679399858270"} 2023-03-21 11:57:38,271 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-03-21 11:57:38,272 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-03-21 11:57:38,273 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679399858273"}]},"ts":"1679399858273"} 2023-03-21 11:57:38,274 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-03-21 11:57:38,277 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=8dbd4df2f9769e1bcf08d64a8aceb9ad, ASSIGN}] 2023-03-21 11:57:38,280 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=8dbd4df2f9769e1bcf08d64a8aceb9ad, ASSIGN 2023-03-21 11:57:38,281 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=8dbd4df2f9769e1bcf08d64a8aceb9ad, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,34369,1679399732030; forceNewPlan=false, retain=false 2023-03-21 11:57:38,433 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=8dbd4df2f9769e1bcf08d64a8aceb9ad, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:38,433 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679399858433"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1679399858433"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679399858433"}]},"ts":"1679399858433"} 2023-03-21 11:57:38,436 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=49, ppid=48, state=RUNNABLE; OpenRegionProcedure 8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030}] 2023-03-21 11:57:38,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-03-21 11:57:38,596 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:38,597 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 8dbd4df2f9769e1bcf08d64a8aceb9ad, NAME => 'TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad.', STARTKEY => '', ENDKEY => ''} 2023-03-21 11:57:38,597 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 8dbd4df2f9769e1bcf08d64a8aceb9ad 2023-03-21 11:57:38,598 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-21 11:57:38,598 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 8dbd4df2f9769e1bcf08d64a8aceb9ad 2023-03-21 11:57:38,598 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 8dbd4df2f9769e1bcf08d64a8aceb9ad 2023-03-21 11:57:38,599 INFO [StoreOpener-8dbd4df2f9769e1bcf08d64a8aceb9ad-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 8dbd4df2f9769e1bcf08d64a8aceb9ad 2023-03-21 11:57:38,600 DEBUG [StoreOpener-8dbd4df2f9769e1bcf08d64a8aceb9ad-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A 2023-03-21 11:57:38,600 DEBUG [StoreOpener-8dbd4df2f9769e1bcf08d64a8aceb9ad-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A 2023-03-21 11:57:38,600 INFO [StoreOpener-8dbd4df2f9769e1bcf08d64a8aceb9ad-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-21 11:57:38,601 INFO [StoreOpener-8dbd4df2f9769e1bcf08d64a8aceb9ad-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 8dbd4df2f9769e1bcf08d64a8aceb9ad columnFamilyName A 2023-03-21 11:57:38,601 INFO [StoreOpener-8dbd4df2f9769e1bcf08d64a8aceb9ad-1] regionserver.HStore(310): Store=8dbd4df2f9769e1bcf08d64a8aceb9ad/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-21 11:57:38,601 INFO [StoreOpener-8dbd4df2f9769e1bcf08d64a8aceb9ad-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 8dbd4df2f9769e1bcf08d64a8aceb9ad 2023-03-21 11:57:38,602 DEBUG [StoreOpener-8dbd4df2f9769e1bcf08d64a8aceb9ad-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B 2023-03-21 11:57:38,602 DEBUG [StoreOpener-8dbd4df2f9769e1bcf08d64a8aceb9ad-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B 2023-03-21 11:57:38,603 INFO [StoreOpener-8dbd4df2f9769e1bcf08d64a8aceb9ad-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-21 11:57:38,603 INFO [StoreOpener-8dbd4df2f9769e1bcf08d64a8aceb9ad-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 8dbd4df2f9769e1bcf08d64a8aceb9ad columnFamilyName B 2023-03-21 11:57:38,603 INFO [StoreOpener-8dbd4df2f9769e1bcf08d64a8aceb9ad-1] regionserver.HStore(310): Store=8dbd4df2f9769e1bcf08d64a8aceb9ad/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-21 11:57:38,603 INFO [StoreOpener-8dbd4df2f9769e1bcf08d64a8aceb9ad-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 8dbd4df2f9769e1bcf08d64a8aceb9ad 2023-03-21 11:57:38,604 DEBUG [StoreOpener-8dbd4df2f9769e1bcf08d64a8aceb9ad-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C 2023-03-21 11:57:38,604 DEBUG [StoreOpener-8dbd4df2f9769e1bcf08d64a8aceb9ad-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C 2023-03-21 11:57:38,605 INFO [StoreOpener-8dbd4df2f9769e1bcf08d64a8aceb9ad-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-21 11:57:38,605 INFO [StoreOpener-8dbd4df2f9769e1bcf08d64a8aceb9ad-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 8dbd4df2f9769e1bcf08d64a8aceb9ad columnFamilyName C 2023-03-21 11:57:38,605 INFO [StoreOpener-8dbd4df2f9769e1bcf08d64a8aceb9ad-1] regionserver.HStore(310): Store=8dbd4df2f9769e1bcf08d64a8aceb9ad/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-21 11:57:38,605 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:38,606 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad 2023-03-21 11:57:38,606 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad 2023-03-21 11:57:38,608 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-21 11:57:38,609 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 8dbd4df2f9769e1bcf08d64a8aceb9ad 2023-03-21 11:57:38,610 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-21 11:57:38,611 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 8dbd4df2f9769e1bcf08d64a8aceb9ad; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67741186, jitterRate=0.009422332048416138}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-21 11:57:38,611 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 8dbd4df2f9769e1bcf08d64a8aceb9ad: 2023-03-21 11:57:38,611 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad., pid=49, masterSystemTime=1679399858588 2023-03-21 11:57:38,612 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:38,612 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:38,612 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=8dbd4df2f9769e1bcf08d64a8aceb9ad, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:38,612 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679399858612"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1679399858612"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1679399858612"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1679399858612"}]},"ts":"1679399858612"} 2023-03-21 11:57:38,614 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=49, resume processing ppid=48 2023-03-21 11:57:38,614 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=49, ppid=48, state=SUCCESS; OpenRegionProcedure 8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 in 178 msec 2023-03-21 11:57:38,615 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=48, resume processing ppid=47 2023-03-21 11:57:38,615 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=48, ppid=47, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=8dbd4df2f9769e1bcf08d64a8aceb9ad, ASSIGN in 337 msec 2023-03-21 11:57:38,615 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-03-21 11:57:38,616 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679399858616"}]},"ts":"1679399858616"} 2023-03-21 11:57:38,616 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-03-21 11:57:38,618 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-03-21 11:57:38,619 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=47, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1840 sec 2023-03-21 11:57:39,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-03-21 11:57:39,546 INFO [Listener at localhost.localdomain/37475] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 47 completed 2023-03-21 11:57:39,547 DEBUG [Listener at localhost.localdomain/37475] zookeeper.ReadOnlyZKClient(139): Connect 0x17a38ae8 to 127.0.0.1:61761 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-21 11:57:39,555 DEBUG [Listener at localhost.localdomain/37475] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5a48c89e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-21 11:57:39,557 DEBUG [Listener at localhost.localdomain/37475] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-21 11:57:39,561 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:48268, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-21 11:57:39,562 DEBUG [Listener at localhost.localdomain/37475] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-03-21 11:57:39,566 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:55220, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-03-21 11:57:39,568 DEBUG [Listener at localhost.localdomain/37475] zookeeper.ReadOnlyZKClient(139): Connect 0x3e21fa39 to 127.0.0.1:61761 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-21 11:57:39,573 DEBUG [Listener at localhost.localdomain/37475] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@355ed3df, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-21 11:57:39,574 DEBUG [Listener at localhost.localdomain/37475] zookeeper.ReadOnlyZKClient(139): Connect 0x7cb49278 to 127.0.0.1:61761 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-21 11:57:39,578 DEBUG [Listener at localhost.localdomain/37475] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4df55fe5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-21 11:57:39,579 DEBUG [Listener at localhost.localdomain/37475] zookeeper.ReadOnlyZKClient(139): Connect 0x50e89dd2 to 127.0.0.1:61761 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-21 11:57:39,582 DEBUG [Listener at localhost.localdomain/37475] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@817e2e5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-21 11:57:39,583 DEBUG [Listener at localhost.localdomain/37475] zookeeper.ReadOnlyZKClient(139): Connect 0x545adde0 to 127.0.0.1:61761 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-21 11:57:39,587 DEBUG [Listener at localhost.localdomain/37475] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4048dcb4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-21 11:57:39,588 DEBUG [Listener at localhost.localdomain/37475] zookeeper.ReadOnlyZKClient(139): Connect 0x4c4d3f2e to 127.0.0.1:61761 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-21 11:57:39,592 DEBUG [Listener at localhost.localdomain/37475] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4e7a5e9a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-21 11:57:39,592 DEBUG [Listener at localhost.localdomain/37475] zookeeper.ReadOnlyZKClient(139): Connect 0x62447900 to 127.0.0.1:61761 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-21 11:57:39,596 DEBUG [Listener at localhost.localdomain/37475] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7ed6755c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-21 11:57:39,596 DEBUG [Listener at localhost.localdomain/37475] zookeeper.ReadOnlyZKClient(139): Connect 0x519db5f2 to 127.0.0.1:61761 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-21 11:57:39,607 DEBUG [Listener at localhost.localdomain/37475] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@10505ab8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-21 11:57:39,608 DEBUG [Listener at localhost.localdomain/37475] zookeeper.ReadOnlyZKClient(139): Connect 0x5c9c1feb to 127.0.0.1:61761 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-21 11:57:39,616 DEBUG [Listener at localhost.localdomain/37475] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@609213af, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-21 11:57:39,616 DEBUG [Listener at localhost.localdomain/37475] zookeeper.ReadOnlyZKClient(139): Connect 0x22bf9ddd to 127.0.0.1:61761 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-21 11:57:39,619 DEBUG [Listener at localhost.localdomain/37475] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1ee0ad26, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-21 11:57:39,620 DEBUG [Listener at localhost.localdomain/37475] zookeeper.ReadOnlyZKClient(139): Connect 0x34c4ae9a to 127.0.0.1:61761 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-21 11:57:39,625 DEBUG [Listener at localhost.localdomain/37475] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6acb8caa, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-21 11:57:39,630 DEBUG [hconnection-0x6a8f25-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-21 11:57:39,630 DEBUG [hconnection-0x4150e0b6-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-21 11:57:39,630 DEBUG [hconnection-0x117641be-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-21 11:57:39,630 DEBUG [hconnection-0xe9b7743-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-21 11:57:39,632 DEBUG [hconnection-0x76d03e3c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-21 11:57:39,632 DEBUG [hconnection-0x72678ea9-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-21 11:57:39,632 DEBUG [hconnection-0x34f40ca3-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-21 11:57:39,635 DEBUG [hconnection-0x178d2f9b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-21 11:57:39,636 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:57:39,632 DEBUG [hconnection-0x26e28ef3-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-21 11:57:39,632 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:48284, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-21 11:57:39,635 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:48302, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-21 11:57:39,634 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:48286, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-21 11:57:39,641 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:48304, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-21 11:57:39,644 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:48324, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-21 11:57:39,644 DEBUG [hconnection-0x70074182-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-21 11:57:39,644 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:48312, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-21 11:57:39,645 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:48336, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-21 11:57:39,649 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:48350, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-21 11:57:39,650 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:48330, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-21 11:57:39,650 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:48362, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-21 11:57:39,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:57:39,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:57:39,656 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:57:39,656 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:39,656 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:57:39,656 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:57:39,657 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:39,657 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:39,658 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:39,658 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:39,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:39,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:39,658 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:39,658 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:57:39,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:39,658 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:39,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:57:39,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:39,660 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:57:39,660 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:57:39,660 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:57:39,660 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:57:39,661 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:39,661 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:57:39,661 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:57:39,661 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:57:39,661 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:39,661 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. started... 2023-03-21 11:57:39,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(9158): Flush requested on 8dbd4df2f9769e1bcf08d64a8aceb9ad 2023-03-21 11:57:39,663 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8dbd4df2f9769e1bcf08d64a8aceb9ad 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-21 11:57:39,663 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dbd4df2f9769e1bcf08d64a8aceb9ad, store=A 2023-03-21 11:57:39,664 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:57:39,664 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dbd4df2f9769e1bcf08d64a8aceb9ad, store=B 2023-03-21 11:57:39,664 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:57:39,664 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dbd4df2f9769e1bcf08d64a8aceb9ad, store=C 2023-03-21 11:57:39,664 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:57:39,666 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. as already flushing 2023-03-21 11:57:39,666 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:39,667 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:57:39,667 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:57:39,667 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:39,669 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:57:39,669 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:57:39,673 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:39,673 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:39,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:39,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:39,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:39,673 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:39,673 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:57:39,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:39,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:39,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:39,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:39,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:39,674 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:57:39,675 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7aec67b1[Count = 0] remaining members to acquire global barrier 2023-03-21 11:57:39,675 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:57:39,675 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:39,675 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:39,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:39,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:39,675 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:57:39,675 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:57:39,675 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:57:39,676 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:39,676 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:57:39,676 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:39,676 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:57:39,676 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:39,676 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:57:39,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:39,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:39,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:39,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:39,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:39,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:39,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:39,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:39,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:39,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:39,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:57:39,678 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:57:39,678 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:57:39,679 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:57:39,679 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:57:39,679 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:57:39,679 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:57:39,679 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:39,679 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:39,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 5 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48324 deadline: 1679399919677, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:39,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48336 deadline: 1679399919677, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:39,679 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:39,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48302 deadline: 1679399919677, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:39,681 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:39,681 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:39,681 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:39,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:39,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:39,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:39,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:39,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:39,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:39,681 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:39,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:39,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:39,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:39,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:39,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:39,682 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:39,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 5 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48362 deadline: 1679399919678, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:39,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:39,682 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:39,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:39,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:39,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:39,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:39,683 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:39,683 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:39,683 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:39,683 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:39,683 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:57:39,683 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:57:39,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:39,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:39,683 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:39,684 DEBUG [Thread-2783] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:57:39,684 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:57:39,683 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:39,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:39,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:39,684 DEBUG [Thread-2783] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:57:39,684 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:39,684 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:39,684 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:39,684 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:39,686 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:39,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48284 deadline: 1679399919679, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:39,784 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:39,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48336 deadline: 1679399919780, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:39,784 DEBUG [Thread-2783] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:39,784 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:39,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48324 deadline: 1679399919781, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:39,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:39,785 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:57:39,786 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:39,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48362 deadline: 1679399919782, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:39,786 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:39,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48302 deadline: 1679399919782, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:39,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:57:39,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:57:39,787 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:57:39,787 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:39,787 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:57:39,787 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:57:39,787 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:39,787 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:39,788 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:39,788 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:39,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:39,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:39,788 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:39,788 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:57:39,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:39,788 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:39,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:57:39,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:39,788 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:57:39,788 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:57:39,789 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:57:39,789 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:57:39,789 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:39,789 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:57:39,790 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:39,790 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:57:39,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48284 deadline: 1679399919787, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:39,790 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:39,790 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:57:39,790 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. started... 2023-03-21 11:57:39,790 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. as already flushing 2023-03-21 11:57:39,790 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:39,790 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:57:39,790 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:57:39,790 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:39,797 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:57:39,797 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:57:39,803 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:39,803 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:39,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:39,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:39,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:39,803 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:39,803 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:57:39,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:39,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:39,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:39,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:39,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:39,804 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:57:39,804 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b4dd2d4[Count = 0] remaining members to acquire global barrier 2023-03-21 11:57:39,804 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:57:39,804 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:39,806 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:39,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:39,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:39,806 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:57:39,807 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:57:39,807 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:57:39,807 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:39,807 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:57:39,807 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:57:39,807 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:39,807 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:39,807 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:57:39,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:39,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:39,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:39,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:39,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:39,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:39,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:39,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:39,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:39,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:39,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:57:39,809 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:57:39,809 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:57:39,809 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:57:39,809 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:57:39,809 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:57:39,809 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:57:39,810 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:39,810 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:39,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:39,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:39,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:39,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:39,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:39,810 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:39,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:39,810 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:39,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:39,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:39,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:39,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:39,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:39,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:39,811 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:39,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:39,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:39,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:39,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:39,813 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:39,813 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:39,813 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:39,813 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:39,813 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:57:39,813 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:57:39,813 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:39,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:39,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:39,813 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:39,813 DEBUG [Thread-2783] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:57:39,813 DEBUG [Thread-2783] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:57:39,813 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:39,813 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:39,813 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:39,813 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:57:39,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:39,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:39,813 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:39,913 DEBUG [Thread-2783] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:39,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:39,915 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:57:39,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:57:39,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:57:39,916 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:57:39,916 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:39,916 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:57:39,916 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:57:39,917 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:39,917 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:39,917 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:39,917 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:39,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:39,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:39,918 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:39,918 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:57:39,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:39,918 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:39,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:57:39,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:39,918 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:57:39,918 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:57:39,918 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:57:39,918 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:57:39,918 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:39,919 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:57:39,919 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:57:39,919 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:57:39,919 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:39,919 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. started... 2023-03-21 11:57:39,919 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. as already flushing 2023-03-21 11:57:39,919 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:39,919 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:57:39,919 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:57:39,919 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:39,919 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:57:39,919 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:57:39,920 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:39,920 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:39,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:39,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:39,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:39,920 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:39,920 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:57:39,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:39,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:39,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:39,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:39,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:39,922 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:57:39,922 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a82a387[Count = 0] remaining members to acquire global barrier 2023-03-21 11:57:39,922 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:57:39,922 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:39,922 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:39,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:39,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:39,922 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:57:39,923 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:57:39,923 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:39,923 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:57:39,923 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:57:39,923 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:39,923 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:57:39,923 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:39,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:39,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:39,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:39,923 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:57:39,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:39,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:39,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:39,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:39,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:39,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:39,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:39,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:57:39,925 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:57:39,925 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:57:39,925 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:57:39,925 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:57:39,925 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:57:39,925 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:57:39,926 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:39,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:39,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:39,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:39,926 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:39,926 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:39,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:39,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:39,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:39,927 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:39,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:39,927 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:39,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:39,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:39,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:39,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:39,928 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:39,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:39,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:39,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:39,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:39,929 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:39,929 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:39,929 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:39,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:39,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:39,929 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:39,929 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:57:39,929 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:57:39,929 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:39,929 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:57:39,929 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:39,929 DEBUG [Thread-2783] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:57:39,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:39,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:39,930 DEBUG [Thread-2783] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:57:39,929 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:39,930 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:39,930 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:39,930 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:39,985 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:39,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48336 deadline: 1679399919985, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:39,991 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:39,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48324 deadline: 1679399919986, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:39,991 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:39,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48362 deadline: 1679399919987, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:39,992 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:39,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48302 deadline: 1679399919988, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:39,992 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:39,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48284 deadline: 1679399919991, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:40,030 DEBUG [Thread-2783] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:40,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:40,031 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:57:40,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:57:40,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:57:40,032 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:57:40,032 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:40,032 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:57:40,032 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:57:40,033 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:40,033 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:40,033 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:40,033 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:40,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:40,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:40,033 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:40,033 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:57:40,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:40,034 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:40,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:57:40,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:40,034 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:57:40,034 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:57:40,034 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:57:40,034 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:57:40,034 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:40,034 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:57:40,035 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:57:40,035 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:57:40,035 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:40,035 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. started... 2023-03-21 11:57:40,035 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. as already flushing 2023-03-21 11:57:40,035 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:40,035 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:57:40,035 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:57:40,035 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:40,035 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:57:40,035 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:57:40,036 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:40,036 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:40,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:40,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:40,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:40,036 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:40,036 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:57:40,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:40,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:40,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:40,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:40,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:40,037 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:57:40,037 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4e7e8958[Count = 0] remaining members to acquire global barrier 2023-03-21 11:57:40,037 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:57:40,037 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:40,037 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:40,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:40,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:40,037 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:40,037 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:57:40,038 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:57:40,037 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:57:40,038 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:57:40,038 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:40,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:40,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:40,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:40,038 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:57:40,038 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:40,038 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:57:40,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:40,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:40,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:40,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:40,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:40,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:40,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:40,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:57:40,040 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:57:40,040 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:57:40,040 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:57:40,040 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:57:40,040 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:57:40,040 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:57:40,041 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:40,041 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:40,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:40,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:40,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:40,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:40,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:40,041 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:40,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:40,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:40,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:40,042 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:40,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:40,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:40,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:40,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:40,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:40,044 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:40,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:40,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:40,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:40,045 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:40,045 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:40,045 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:40,045 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:40,045 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:40,045 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:57:40,045 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:57:40,045 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:40,045 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:40,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:40,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:40,045 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:40,045 DEBUG [Thread-2783] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:57:40,045 DEBUG [Thread-2783] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:57:40,045 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:40,046 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:40,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:40,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:40,073 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=13 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/A/48f70e5740e24bf2b2ce7bf8b98eea79 2023-03-21 11:57:40,083 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=13 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/B/efdec08f12ce4e268b133e793464c267 2023-03-21 11:57:40,091 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=13 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/C/3ad32817e68d4feda55ff068caa3304d 2023-03-21 11:57:40,094 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/A/48f70e5740e24bf2b2ce7bf8b98eea79 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/48f70e5740e24bf2b2ce7bf8b98eea79 2023-03-21 11:57:40,097 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/48f70e5740e24bf2b2ce7bf8b98eea79, entries=150, sequenceid=13, filesize=11.6 K 2023-03-21 11:57:40,097 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/B/efdec08f12ce4e268b133e793464c267 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/efdec08f12ce4e268b133e793464c267 2023-03-21 11:57:40,100 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/efdec08f12ce4e268b133e793464c267, entries=150, sequenceid=13, filesize=11.6 K 2023-03-21 11:57:40,100 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/C/3ad32817e68d4feda55ff068caa3304d as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/3ad32817e68d4feda55ff068caa3304d 2023-03-21 11:57:40,103 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/3ad32817e68d4feda55ff068caa3304d, entries=150, sequenceid=13, filesize=11.6 K 2023-03-21 11:57:40,103 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 8dbd4df2f9769e1bcf08d64a8aceb9ad in 440ms, sequenceid=13, compaction requested=false 2023-03-21 11:57:40,104 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8dbd4df2f9769e1bcf08d64a8aceb9ad: 2023-03-21 11:57:40,146 DEBUG [Thread-2783] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:40,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:40,147 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:57:40,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:57:40,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:57:40,148 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:57:40,148 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:40,148 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:57:40,148 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:57:40,148 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:40,148 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:40,149 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:40,149 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:40,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:40,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:40,149 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:40,149 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:57:40,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:40,149 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:40,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:57:40,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:40,149 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:57:40,150 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:57:40,150 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:57:40,150 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:57:40,150 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:40,150 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:57:40,150 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:57:40,150 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:57:40,150 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:40,150 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. started... 2023-03-21 11:57:40,150 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8dbd4df2f9769e1bcf08d64a8aceb9ad 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-21 11:57:40,150 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dbd4df2f9769e1bcf08d64a8aceb9ad, store=A 2023-03-21 11:57:40,150 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:57:40,150 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dbd4df2f9769e1bcf08d64a8aceb9ad, store=B 2023-03-21 11:57:40,151 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:57:40,151 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dbd4df2f9769e1bcf08d64a8aceb9ad, store=C 2023-03-21 11:57:40,151 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:57:40,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(9158): Flush requested on 8dbd4df2f9769e1bcf08d64a8aceb9ad 2023-03-21 11:57:40,291 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. as already flushing 2023-03-21 11:57:40,306 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:40,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48284 deadline: 1679399920299, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:40,306 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:40,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48302 deadline: 1679399920300, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:40,306 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:40,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48336 deadline: 1679399920300, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:40,306 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:40,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48324 deadline: 1679399920302, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:40,307 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:40,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48362 deadline: 1679399920302, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:40,409 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:40,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48284 deadline: 1679399920407, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:40,409 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:40,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48302 deadline: 1679399920407, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:40,410 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:40,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48336 deadline: 1679399920407, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:40,410 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:40,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48324 deadline: 1679399920407, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:40,410 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:40,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48362 deadline: 1679399920408, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:40,557 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=33 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/A/7a7ab837338a4b68bac517fbb992affa 2023-03-21 11:57:40,613 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:40,613 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:40,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48302 deadline: 1679399920610, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:40,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48284 deadline: 1679399920610, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:40,614 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:40,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48362 deadline: 1679399920610, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:40,614 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:40,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48336 deadline: 1679399920611, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:40,614 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:40,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48324 deadline: 1679399920611, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:40,918 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:40,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48336 deadline: 1679399920915, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:40,919 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:40,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48324 deadline: 1679399920915, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:40,919 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:40,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48302 deadline: 1679399920915, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:40,919 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:40,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48284 deadline: 1679399920915, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:40,919 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:40,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48362 deadline: 1679399920916, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:40,965 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=33 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/B/52168d3b40b74cde9f216a960c66fab0 2023-03-21 11:57:40,973 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=33 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/C/ba5820a8f0f045988e1819b794743b47 2023-03-21 11:57:40,975 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/A/7a7ab837338a4b68bac517fbb992affa as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/7a7ab837338a4b68bac517fbb992affa 2023-03-21 11:57:40,978 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/7a7ab837338a4b68bac517fbb992affa, entries=150, sequenceid=33, filesize=11.6 K 2023-03-21 11:57:40,978 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/B/52168d3b40b74cde9f216a960c66fab0 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/52168d3b40b74cde9f216a960c66fab0 2023-03-21 11:57:40,981 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/52168d3b40b74cde9f216a960c66fab0, entries=150, sequenceid=33, filesize=11.6 K 2023-03-21 11:57:40,981 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/C/ba5820a8f0f045988e1819b794743b47 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/ba5820a8f0f045988e1819b794743b47 2023-03-21 11:57:40,985 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/ba5820a8f0f045988e1819b794743b47, entries=150, sequenceid=33, filesize=11.6 K 2023-03-21 11:57:40,986 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 8dbd4df2f9769e1bcf08d64a8aceb9ad in 836ms, sequenceid=33, compaction requested=false 2023-03-21 11:57:40,986 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8dbd4df2f9769e1bcf08d64a8aceb9ad: 2023-03-21 11:57:40,986 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:40,986 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:57:40,986 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:57:40,986 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:40,986 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:57:40,986 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:57:40,987 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:40,987 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:40,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:40,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:40,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:40,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:40,987 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:40,987 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:57:40,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:40,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:40,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:40,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:40,988 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:57:40,988 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@57b6aec9[Count = 0] remaining members to acquire global barrier 2023-03-21 11:57:40,988 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:57:40,988 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:40,989 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:40,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:40,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:40,989 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:57:40,989 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:40,989 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:57:40,989 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:57:40,989 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:57:40,990 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:40,990 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:57:40,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:40,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:40,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:40,990 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:40,990 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:57:40,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:40,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:40,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:40,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:40,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:40,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:40,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:40,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:57:40,991 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:57:40,991 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:57:40,991 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:57:40,991 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:57:40,991 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:57:40,992 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:57:40,992 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:40,992 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:40,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:40,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:40,992 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:40,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:40,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:40,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:40,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:40,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:40,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:40,993 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:40,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:40,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:40,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:40,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:40,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:40,994 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:40,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:40,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:40,995 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:40,995 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:40,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:40,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:40,995 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:40,995 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:57:40,995 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:57:40,995 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:40,995 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:40,995 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:40,995 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:40,995 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:40,995 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:40,995 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:40,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:40,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:40,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:40,995 DEBUG [Thread-2783] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:57:40,996 DEBUG [Thread-2783] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:57:40,996 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:57:41,096 DEBUG [Thread-2783] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:41,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:41,097 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:57:41,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:57:41,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:57:41,098 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:57:41,099 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:41,099 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:57:41,099 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:57:41,099 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:41,099 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:41,100 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:41,100 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:41,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:41,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:41,100 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:41,100 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:57:41,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:41,100 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:41,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:57:41,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:41,101 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:57:41,101 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:57:41,101 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:57:41,101 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:57:41,101 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:41,101 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:57:41,101 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:57:41,101 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:57:41,101 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:41,101 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. started... 2023-03-21 11:57:41,101 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8dbd4df2f9769e1bcf08d64a8aceb9ad 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-21 11:57:41,101 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dbd4df2f9769e1bcf08d64a8aceb9ad, store=A 2023-03-21 11:57:41,101 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:57:41,101 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dbd4df2f9769e1bcf08d64a8aceb9ad, store=B 2023-03-21 11:57:41,101 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:57:41,101 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dbd4df2f9769e1bcf08d64a8aceb9ad, store=C 2023-03-21 11:57:41,101 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:57:41,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(9158): Flush requested on 8dbd4df2f9769e1bcf08d64a8aceb9ad 2023-03-21 11:57:41,424 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. as already flushing 2023-03-21 11:57:41,442 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:41,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48284 deadline: 1679399921437, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:41,443 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679399861430, max timestamp=1679399861441Num uniques 859; 2023-03-21 11:57:41,443 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679399861430, max timestamp=1679399861441Num uniques 859; 2023-03-21 11:57:41,443 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-21 11:57:41,443 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679399861430, max timestamp=1679399861441Num uniques 859; 2023-03-21 11:57:41,443 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-21 11:57:41,443 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-21 11:57:41,469 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:41,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48324 deadline: 1679399921461, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:41,469 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:41,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48336 deadline: 1679399921463, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:41,472 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:41,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48302 deadline: 1679399921464, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:41,475 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:41,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48362 deadline: 1679399921467, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:41,508 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=45 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/A/46401f06a61b4f9c9189fc6dacf34687 2023-03-21 11:57:41,550 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:41,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48284 deadline: 1679399921543, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:41,577 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:41,577 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:41,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48336 deadline: 1679399921570, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:41,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48324 deadline: 1679399921570, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:41,577 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:41,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48302 deadline: 1679399921573, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:41,583 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:41,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48362 deadline: 1679399921576, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:41,753 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:41,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48284 deadline: 1679399921752, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:41,781 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:41,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48324 deadline: 1679399921779, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:41,784 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:41,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48336 deadline: 1679399921779, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:41,784 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:41,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48302 deadline: 1679399921779, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:41,787 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:41,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48362 deadline: 1679399921783, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:41,916 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=45 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/B/1848cf8a974748bcbfe3f9f7e952b709 2023-03-21 11:57:42,064 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:42,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48284 deadline: 1679399922055, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:42,088 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:42,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48324 deadline: 1679399922083, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:42,090 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:42,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48336 deadline: 1679399922085, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:42,091 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:42,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48302 deadline: 1679399922086, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:42,094 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:42,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48362 deadline: 1679399922089, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:42,210 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-03-21 11:57:42,326 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=45 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/C/c2558cdf5d6b4f219c5d7890dc754e3f 2023-03-21 11:57:42,330 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/A/46401f06a61b4f9c9189fc6dacf34687 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/46401f06a61b4f9c9189fc6dacf34687 2023-03-21 11:57:42,333 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/46401f06a61b4f9c9189fc6dacf34687, entries=150, sequenceid=45, filesize=11.6 K 2023-03-21 11:57:42,333 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/B/1848cf8a974748bcbfe3f9f7e952b709 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/1848cf8a974748bcbfe3f9f7e952b709 2023-03-21 11:57:42,336 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/1848cf8a974748bcbfe3f9f7e952b709, entries=150, sequenceid=45, filesize=11.6 K 2023-03-21 11:57:42,336 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/C/c2558cdf5d6b4f219c5d7890dc754e3f as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/c2558cdf5d6b4f219c5d7890dc754e3f 2023-03-21 11:57:42,339 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/c2558cdf5d6b4f219c5d7890dc754e3f, entries=150, sequenceid=45, filesize=11.6 K 2023-03-21 11:57:42,340 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=120.76 KB/123660 for 8dbd4df2f9769e1bcf08d64a8aceb9ad in 1239ms, sequenceid=45, compaction requested=true 2023-03-21 11:57:42,340 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8dbd4df2f9769e1bcf08d64a8aceb9ad: 2023-03-21 11:57:42,340 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:42,340 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:57:42,340 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:57:42,340 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:42,340 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:57:42,340 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:57:42,342 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:42,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:42,342 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:42,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:42,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:42,342 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:42,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:42,342 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:57:42,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:42,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:42,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:42,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:42,344 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:57:42,344 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@42781d99[Count = 0] remaining members to acquire global barrier 2023-03-21 11:57:42,344 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:57:42,344 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:42,344 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:42,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:42,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:42,344 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:57:42,344 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:57:42,344 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:57:42,344 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:42,345 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:57:42,345 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:57:42,345 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:42,345 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:42,345 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:57:42,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:42,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:42,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:42,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:42,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:42,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:42,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:42,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:42,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:42,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:42,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:57:42,348 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:57:42,348 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:57:42,348 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:57:42,348 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:57:42,348 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:57:42,348 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:57:42,349 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:42,349 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:42,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:42,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:42,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:42,349 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:42,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:42,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:42,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:42,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:42,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:42,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:42,350 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:42,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:42,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:42,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:42,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:42,351 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:42,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:42,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:42,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:42,353 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:42,354 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:42,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:42,354 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:42,354 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:42,354 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:57:42,354 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:57:42,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:42,354 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:42,354 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:42,354 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:42,354 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:42,354 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:42,354 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:42,354 DEBUG [Thread-2783] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:57:42,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:42,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:42,354 DEBUG [Thread-2783] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:57:42,454 DEBUG [Thread-2783] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:42,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:42,456 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:57:42,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:57:42,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:57:42,457 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:57:42,457 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:42,457 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:57:42,457 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:57:42,457 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:42,458 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:42,458 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:42,458 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:42,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:42,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:42,458 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:42,458 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:57:42,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:42,459 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:42,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:57:42,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:42,459 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:57:42,459 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:57:42,459 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:57:42,459 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:57:42,459 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:42,459 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:57:42,459 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:57:42,459 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:57:42,459 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:42,460 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. started... 2023-03-21 11:57:42,460 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8dbd4df2f9769e1bcf08d64a8aceb9ad 3/3 column families, dataSize=120.76 KB heapSize=340.45 KB 2023-03-21 11:57:42,460 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dbd4df2f9769e1bcf08d64a8aceb9ad, store=A 2023-03-21 11:57:42,460 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-21 11:57:42,460 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dbd4df2f9769e1bcf08d64a8aceb9ad, store=B 2023-03-21 11:57:42,460 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-21 11:57:42,460 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dbd4df2f9769e1bcf08d64a8aceb9ad, store=C 2023-03-21 11:57:42,460 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-21 11:57:42,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(9158): Flush requested on 8dbd4df2f9769e1bcf08d64a8aceb9ad 2023-03-21 11:57:42,570 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. as already flushing 2023-03-21 11:57:42,604 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:42,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48362 deadline: 1679399922596, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:42,605 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:42,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48302 deadline: 1679399922598, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:42,611 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:42,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48284 deadline: 1679399922599, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:42,611 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:42,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48324 deadline: 1679399922600, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:42,611 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:42,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48336 deadline: 1679399922601, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:42,711 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:42,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48302 deadline: 1679399922706, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:42,716 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:42,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48284 deadline: 1679399922712, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:42,716 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:42,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48324 deadline: 1679399922712, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:42,717 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:42,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48336 deadline: 1679399922712, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:42,867 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=79 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/A/0e54f1c8c20443339da361430aebe1fa 2023-03-21 11:57:42,915 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:42,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48302 deadline: 1679399922912, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:42,924 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:42,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48324 deadline: 1679399922917, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:42,924 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:42,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48284 deadline: 1679399922917, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:42,924 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:42,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48336 deadline: 1679399922917, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:43,223 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:43,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48302 deadline: 1679399923217, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:43,228 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:43,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48284 deadline: 1679399923225, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:43,231 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:43,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48324 deadline: 1679399923226, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:43,231 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:43,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48336 deadline: 1679399923227, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:43,275 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=79 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/B/ee565f7fbdee45298a18bef5464d4206 2023-03-21 11:57:43,283 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=79 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/C/c59afd9df3de4dad85f72d1caa39551f 2023-03-21 11:57:43,286 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/A/0e54f1c8c20443339da361430aebe1fa as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/0e54f1c8c20443339da361430aebe1fa 2023-03-21 11:57:43,288 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/0e54f1c8c20443339da361430aebe1fa, entries=150, sequenceid=79, filesize=11.6 K 2023-03-21 11:57:43,288 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/B/ee565f7fbdee45298a18bef5464d4206 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/ee565f7fbdee45298a18bef5464d4206 2023-03-21 11:57:43,291 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/ee565f7fbdee45298a18bef5464d4206, entries=150, sequenceid=79, filesize=11.6 K 2023-03-21 11:57:43,292 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/C/c59afd9df3de4dad85f72d1caa39551f as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/c59afd9df3de4dad85f72d1caa39551f 2023-03-21 11:57:43,294 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/c59afd9df3de4dad85f72d1caa39551f, entries=150, sequenceid=79, filesize=11.6 K 2023-03-21 11:57:43,295 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~340.41 KB/348576, currentSize=60.38 KB/61830 for 8dbd4df2f9769e1bcf08d64a8aceb9ad in 835ms, sequenceid=79, compaction requested=true 2023-03-21 11:57:43,295 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8dbd4df2f9769e1bcf08d64a8aceb9ad: 2023-03-21 11:57:43,295 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:43,295 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:57:43,295 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:57:43,295 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:43,295 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:57:43,295 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:57:43,296 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:43,296 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:43,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:43,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:43,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:43,297 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:43,297 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:57:43,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:43,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:43,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:43,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:43,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:43,298 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:57:43,298 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d8900ca[Count = 0] remaining members to acquire global barrier 2023-03-21 11:57:43,298 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:57:43,298 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:43,298 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:43,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:43,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:43,298 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:57:43,298 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:57:43,298 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:57:43,298 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:43,298 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:57:43,299 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:43,299 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:57:43,299 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:43,299 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:57:43,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:43,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:43,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:43,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:43,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:43,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:43,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:43,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:43,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:43,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:43,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:57:43,301 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:57:43,301 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:57:43,301 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:57:43,301 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:57:43,301 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:57:43,301 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:57:43,302 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:43,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:43,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:43,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:43,302 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:43,302 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:43,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:43,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:43,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:43,303 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:43,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:43,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:43,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:43,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:43,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:43,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:43,304 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:43,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:43,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:43,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:43,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:43,305 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:43,305 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:43,305 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:43,305 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:43,305 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:43,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:43,305 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:57:43,305 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:57:43,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:43,305 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:43,305 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:43,305 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:43,305 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:57:43,305 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:43,305 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:43,305 DEBUG [Thread-2783] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:57:43,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:43,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:43,305 DEBUG [Thread-2783] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:57:43,405 DEBUG [Thread-2783] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:43,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:43,407 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:57:43,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:57:43,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:57:43,408 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:57:43,408 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:43,408 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:57:43,408 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:57:43,408 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:43,408 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:43,410 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:43,410 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:43,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:43,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:43,410 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:43,410 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:57:43,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:43,410 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:43,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:57:43,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:43,410 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:57:43,411 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:57:43,411 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:57:43,411 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:43,411 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:57:43,411 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:57:43,411 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:57:43,411 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:43,411 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. started... 2023-03-21 11:57:43,411 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8dbd4df2f9769e1bcf08d64a8aceb9ad 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-21 11:57:43,411 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dbd4df2f9769e1bcf08d64a8aceb9ad, store=A 2023-03-21 11:57:43,411 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:57:43,411 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dbd4df2f9769e1bcf08d64a8aceb9ad, store=B 2023-03-21 11:57:43,411 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:57:43,411 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dbd4df2f9769e1bcf08d64a8aceb9ad, store=C 2023-03-21 11:57:43,412 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:57:43,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(9158): Flush requested on 8dbd4df2f9769e1bcf08d64a8aceb9ad 2023-03-21 11:57:43,607 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. as already flushing 2023-03-21 11:57:43,686 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:43,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48362 deadline: 1679399923684, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:43,729 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:43,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48302 deadline: 1679399923725, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:43,735 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:43,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48284 deadline: 1679399923731, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:43,739 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:43,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48336 deadline: 1679399923734, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:43,740 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:43,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48324 deadline: 1679399923735, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:43,790 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:43,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48362 deadline: 1679399923787, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:43,817 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=91 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/A/bb0436b397b24ef58e44d776c40f73bc 2023-03-21 11:57:43,826 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=91 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/B/dc37a1f2797d499ea1d6e8aa8a1367e4 2023-03-21 11:57:43,834 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=91 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/C/f43be7b7323c479c98d2ce6f2254d49d 2023-03-21 11:57:43,838 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/A/bb0436b397b24ef58e44d776c40f73bc as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/bb0436b397b24ef58e44d776c40f73bc 2023-03-21 11:57:43,841 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/bb0436b397b24ef58e44d776c40f73bc, entries=150, sequenceid=91, filesize=11.6 K 2023-03-21 11:57:43,841 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/B/dc37a1f2797d499ea1d6e8aa8a1367e4 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/dc37a1f2797d499ea1d6e8aa8a1367e4 2023-03-21 11:57:43,844 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/dc37a1f2797d499ea1d6e8aa8a1367e4, entries=150, sequenceid=91, filesize=11.6 K 2023-03-21 11:57:43,845 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/C/f43be7b7323c479c98d2ce6f2254d49d as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/f43be7b7323c479c98d2ce6f2254d49d 2023-03-21 11:57:43,847 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/f43be7b7323c479c98d2ce6f2254d49d, entries=150, sequenceid=91, filesize=11.6 K 2023-03-21 11:57:43,848 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 8dbd4df2f9769e1bcf08d64a8aceb9ad in 437ms, sequenceid=91, compaction requested=true 2023-03-21 11:57:43,848 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8dbd4df2f9769e1bcf08d64a8aceb9ad: 2023-03-21 11:57:43,848 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:43,848 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:57:43,848 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:57:43,848 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:43,848 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:57:43,848 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:57:43,849 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:43,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:43,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:43,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:43,849 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:43,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:43,850 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:43,850 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:57:43,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:43,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:43,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:43,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:43,851 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:57:43,851 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1787686a[Count = 0] remaining members to acquire global barrier 2023-03-21 11:57:43,851 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:57:43,851 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:43,851 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:43,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:43,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:43,851 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:43,852 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:57:43,852 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:57:43,852 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:57:43,852 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:57:43,852 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:43,852 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:57:43,852 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:43,852 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:57:43,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:43,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:43,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:43,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:43,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:43,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:43,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:43,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:43,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:43,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:43,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:57:43,854 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:57:43,854 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:57:43,854 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:57:43,854 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:57:43,854 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:57:43,854 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:57:43,855 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:43,855 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:43,855 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:43,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:43,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:43,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:43,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:43,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:43,855 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:43,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:43,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:43,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:43,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:43,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:43,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:43,856 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:43,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:43,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:43,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:43,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:43,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:43,857 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:43,857 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:43,857 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:43,857 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:43,857 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:43,857 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:43,857 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:57:43,857 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:57:43,858 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:57:43,857 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:43,857 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:43,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:43,858 DEBUG [Thread-2783] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:57:43,857 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:43,858 DEBUG [Thread-2783] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:57:43,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:43,858 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:43,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:43,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:43,958 DEBUG [Thread-2783] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:43,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:43,959 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:57:43,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:57:43,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:57:43,961 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:57:43,961 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:43,961 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:57:43,961 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:57:43,961 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:43,961 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:43,962 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:43,962 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:43,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:43,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:43,962 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:43,962 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:57:43,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:43,962 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:43,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:57:43,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:43,962 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:57:43,962 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:57:43,963 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:57:43,963 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:57:43,963 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:43,963 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:57:43,963 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:57:43,963 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:57:43,963 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:43,963 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. started... 2023-03-21 11:57:43,963 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8dbd4df2f9769e1bcf08d64a8aceb9ad 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-21 11:57:43,963 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dbd4df2f9769e1bcf08d64a8aceb9ad, store=A 2023-03-21 11:57:43,963 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:57:43,963 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dbd4df2f9769e1bcf08d64a8aceb9ad, store=B 2023-03-21 11:57:43,963 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:57:43,963 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dbd4df2f9769e1bcf08d64a8aceb9ad, store=C 2023-03-21 11:57:43,963 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:57:43,971 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=111 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/A/d18ce65c0770425b9f9bf47b5f20fe8a 2023-03-21 11:57:43,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(9158): Flush requested on 8dbd4df2f9769e1bcf08d64a8aceb9ad 2023-03-21 11:57:44,000 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. as already flushing 2023-03-21 11:57:44,048 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:44,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48362 deadline: 1679399924048, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:44,150 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:44,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48362 deadline: 1679399924149, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:44,354 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:44,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48362 deadline: 1679399924352, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:44,379 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=111 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/B/a53e2fa6af314745818d764952284452 2023-03-21 11:57:44,658 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:44,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48362 deadline: 1679399924656, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:44,735 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:44,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48302 deadline: 1679399924734, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:44,745 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:44,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48284 deadline: 1679399924736, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:44,745 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:44,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48336 deadline: 1679399924741, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:44,751 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:44,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48324 deadline: 1679399924745, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:44,786 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=111 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/C/06cdbe75ba0642bcac8ccdaa2114def6 2023-03-21 11:57:44,789 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/A/d18ce65c0770425b9f9bf47b5f20fe8a as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/d18ce65c0770425b9f9bf47b5f20fe8a 2023-03-21 11:57:44,792 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/d18ce65c0770425b9f9bf47b5f20fe8a, entries=150, sequenceid=111, filesize=11.6 K 2023-03-21 11:57:44,793 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/B/a53e2fa6af314745818d764952284452 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/a53e2fa6af314745818d764952284452 2023-03-21 11:57:44,796 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/a53e2fa6af314745818d764952284452, entries=150, sequenceid=111, filesize=11.6 K 2023-03-21 11:57:44,796 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/C/06cdbe75ba0642bcac8ccdaa2114def6 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/06cdbe75ba0642bcac8ccdaa2114def6 2023-03-21 11:57:44,799 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/06cdbe75ba0642bcac8ccdaa2114def6, entries=150, sequenceid=111, filesize=11.6 K 2023-03-21 11:57:44,799 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 8dbd4df2f9769e1bcf08d64a8aceb9ad in 836ms, sequenceid=111, compaction requested=true 2023-03-21 11:57:44,800 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8dbd4df2f9769e1bcf08d64a8aceb9ad: 2023-03-21 11:57:44,800 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:44,800 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:57:44,800 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:57:44,800 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:44,800 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:57:44,800 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:57:44,801 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:44,801 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:44,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:44,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:44,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:44,801 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:44,801 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:57:44,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:44,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:44,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:44,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:44,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:44,802 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:57:44,802 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@75255cb5[Count = 0] remaining members to acquire global barrier 2023-03-21 11:57:44,802 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:57:44,802 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:44,803 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:44,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:44,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:44,803 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:57:44,803 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:57:44,803 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:57:44,804 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:44,804 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:57:44,804 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:57:44,804 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:57:44,804 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:57:44,804 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:57:44,804 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:57:44,804 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:57:44,805 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:44,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:44,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:44,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:44,805 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:44,805 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:44,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:44,805 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:57:44,805 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:44,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:44,805 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:57:44,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:44,806 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:44,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:44,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:44,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:44,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:44,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:44,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:44,807 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:44,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:44,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:44,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:44,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:44,808 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:44,808 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:44,808 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:44,808 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:57:44,808 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:57:44,808 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:57:44,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:44,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:44,808 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:44,808 DEBUG [Thread-2783] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:57:44,808 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:44,808 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:44,808 DEBUG [Thread-2783] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:57:44,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:44,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:44,809 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:44,809 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:44,809 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:44,809 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:44,909 DEBUG [Thread-2783] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:44,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:44,910 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:57:44,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:57:44,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:57:44,911 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:57:44,911 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:44,911 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:57:44,911 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:57:44,912 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:44,912 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:44,912 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:44,912 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:44,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:44,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:44,913 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:44,913 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:57:44,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:44,913 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:44,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:57:44,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:44,913 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:57:44,913 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:57:44,913 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:57:44,913 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:57:44,913 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:44,913 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:57:44,914 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:57:44,914 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:57:44,914 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:44,914 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. started... 2023-03-21 11:57:44,914 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8dbd4df2f9769e1bcf08d64a8aceb9ad 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-21 11:57:44,914 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dbd4df2f9769e1bcf08d64a8aceb9ad, store=A 2023-03-21 11:57:44,914 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:57:44,914 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dbd4df2f9769e1bcf08d64a8aceb9ad, store=B 2023-03-21 11:57:44,914 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:57:44,914 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dbd4df2f9769e1bcf08d64a8aceb9ad, store=C 2023-03-21 11:57:44,914 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:57:45,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(9158): Flush requested on 8dbd4df2f9769e1bcf08d64a8aceb9ad 2023-03-21 11:57:45,167 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. as already flushing 2023-03-21 11:57:45,265 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:45,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48362 deadline: 1679399925257, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:45,319 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=123 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/A/118e5d59e8a04a8bb6610dae3e71dc70 2023-03-21 11:57:45,368 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:45,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48362 deadline: 1679399925367, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:45,572 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:45,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48362 deadline: 1679399925569, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:45,738 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=123 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/B/0e62ed6012624fa889f1cbd819384503 2023-03-21 11:57:45,879 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:45,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48362 deadline: 1679399925873, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:46,153 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=123 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/C/f16c7a152162417982b660dc6ae0f1c5 2023-03-21 11:57:46,157 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/A/118e5d59e8a04a8bb6610dae3e71dc70 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/118e5d59e8a04a8bb6610dae3e71dc70 2023-03-21 11:57:46,160 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/118e5d59e8a04a8bb6610dae3e71dc70, entries=150, sequenceid=123, filesize=11.6 K 2023-03-21 11:57:46,160 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/B/0e62ed6012624fa889f1cbd819384503 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/0e62ed6012624fa889f1cbd819384503 2023-03-21 11:57:46,163 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/0e62ed6012624fa889f1cbd819384503, entries=150, sequenceid=123, filesize=11.6 K 2023-03-21 11:57:46,163 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/C/f16c7a152162417982b660dc6ae0f1c5 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/f16c7a152162417982b660dc6ae0f1c5 2023-03-21 11:57:46,167 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/f16c7a152162417982b660dc6ae0f1c5, entries=150, sequenceid=123, filesize=11.6 K 2023-03-21 11:57:46,168 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 8dbd4df2f9769e1bcf08d64a8aceb9ad in 1254ms, sequenceid=123, compaction requested=true 2023-03-21 11:57:46,168 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8dbd4df2f9769e1bcf08d64a8aceb9ad: 2023-03-21 11:57:46,169 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:46,169 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:57:46,169 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:57:46,169 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:46,169 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:57:46,170 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:57:46,172 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:46,172 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:46,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:46,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:46,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:46,172 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:46,172 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:57:46,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:46,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:46,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:46,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:46,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:46,173 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:57:46,174 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@77b3035d[Count = 0] remaining members to acquire global barrier 2023-03-21 11:57:46,174 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:57:46,174 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:46,174 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:46,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:46,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:46,174 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:57:46,174 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:57:46,175 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:57:46,175 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:46,175 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:57:46,175 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:57:46,175 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:46,175 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:57:46,176 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:46,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:46,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:46,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:46,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:46,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:46,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:46,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:46,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:46,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:46,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:46,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:57:46,177 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:57:46,177 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:57:46,177 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:57:46,177 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:57:46,177 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:57:46,178 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:57:46,178 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:46,178 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:46,178 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:46,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:46,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:46,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:46,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:46,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:46,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:46,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:46,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:46,179 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:46,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:46,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:46,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:46,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:46,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:46,180 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:46,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:46,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:46,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:46,182 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:46,182 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:46,182 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:46,182 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:57:46,182 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:57:46,182 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:57:46,182 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:46,182 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:46,182 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:46,182 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:46,182 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:46,182 DEBUG [Thread-2783] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:57:46,182 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:46,182 DEBUG [Thread-2783] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:57:46,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:46,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:46,182 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:46,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:46,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:46,282 DEBUG [Thread-2783] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:46,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:46,284 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:57:46,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:57:46,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:57:46,285 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:57:46,285 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:46,285 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:57:46,285 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:57:46,285 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:46,285 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:46,286 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:46,286 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:46,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:46,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:46,286 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:46,286 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:57:46,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:46,287 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:46,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:57:46,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:46,287 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:57:46,287 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:57:46,287 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:57:46,287 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:57:46,287 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:46,287 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:57:46,287 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:57:46,287 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:57:46,287 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:46,287 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. started... 2023-03-21 11:57:46,288 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8dbd4df2f9769e1bcf08d64a8aceb9ad 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-21 11:57:46,288 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dbd4df2f9769e1bcf08d64a8aceb9ad, store=A 2023-03-21 11:57:46,288 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:57:46,288 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dbd4df2f9769e1bcf08d64a8aceb9ad, store=B 2023-03-21 11:57:46,288 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:57:46,288 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dbd4df2f9769e1bcf08d64a8aceb9ad, store=C 2023-03-21 11:57:46,288 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:57:46,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(9158): Flush requested on 8dbd4df2f9769e1bcf08d64a8aceb9ad 2023-03-21 11:57:46,387 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. as already flushing 2023-03-21 11:57:46,449 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:46,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48362 deadline: 1679399926445, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:46,556 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:46,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48362 deadline: 1679399926550, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:46,696 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=143 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/A/93fb6dc50f6543aab2b0b2cb1426906b 2023-03-21 11:57:46,741 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:46,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48302 deadline: 1679399926737, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:46,743 DEBUG [Thread-2775] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4143 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad., hostname=jenkins-hbase20.apache.org,34369,1679399732030, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-21 11:57:46,758 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:46,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48284 deadline: 1679399926753, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:46,760 DEBUG [Thread-2773] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4159 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad., hostname=jenkins-hbase20.apache.org,34369,1679399732030, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-21 11:57:46,765 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:46,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48362 deadline: 1679399926758, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:46,767 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:46,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48336 deadline: 1679399926762, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:46,769 DEBUG [Thread-2779] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4167 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad., hostname=jenkins-hbase20.apache.org,34369,1679399732030, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-21 11:57:46,772 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:46,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48324 deadline: 1679399926764, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:46,774 DEBUG [Thread-2781] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4173 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad., hostname=jenkins-hbase20.apache.org,34369,1679399732030, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-21 11:57:47,069 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:47,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48362 deadline: 1679399927066, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:47,108 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=143 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/B/da5e172d03fd4ff2822262d9d22d3ea4 2023-03-21 11:57:47,516 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=143 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/C/b962320e8ae24525bed2262267651f53 2023-03-21 11:57:47,519 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/A/93fb6dc50f6543aab2b0b2cb1426906b as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/93fb6dc50f6543aab2b0b2cb1426906b 2023-03-21 11:57:47,522 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/93fb6dc50f6543aab2b0b2cb1426906b, entries=150, sequenceid=143, filesize=11.7 K 2023-03-21 11:57:47,523 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/B/da5e172d03fd4ff2822262d9d22d3ea4 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/da5e172d03fd4ff2822262d9d22d3ea4 2023-03-21 11:57:47,525 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/da5e172d03fd4ff2822262d9d22d3ea4, entries=150, sequenceid=143, filesize=11.7 K 2023-03-21 11:57:47,526 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/C/b962320e8ae24525bed2262267651f53 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/b962320e8ae24525bed2262267651f53 2023-03-21 11:57:47,529 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/b962320e8ae24525bed2262267651f53, entries=150, sequenceid=143, filesize=11.7 K 2023-03-21 11:57:47,531 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 8dbd4df2f9769e1bcf08d64a8aceb9ad in 1243ms, sequenceid=143, compaction requested=true 2023-03-21 11:57:47,531 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8dbd4df2f9769e1bcf08d64a8aceb9ad: 2023-03-21 11:57:47,531 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:47,531 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:57:47,531 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:57:47,531 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:47,531 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:57:47,531 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:57:47,532 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:47,532 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:47,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:47,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:47,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:47,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:47,532 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:47,532 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:57:47,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:47,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:47,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:47,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:47,533 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:57:47,533 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@12d9c231[Count = 0] remaining members to acquire global barrier 2023-03-21 11:57:47,533 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:57:47,533 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:47,534 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:47,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:47,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:47,534 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:57:47,534 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:57:47,534 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:57:47,534 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:47,534 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:57:47,535 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:47,535 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:57:47,535 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:47,535 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:57:47,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:47,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:47,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:47,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:47,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:47,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:47,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:47,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:47,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:47,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:47,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:57:47,537 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:57:47,537 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:57:47,537 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:57:47,537 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:57:47,537 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:57:47,537 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:57:47,538 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:47,538 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:47,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:47,538 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:47,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:47,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:47,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:47,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:47,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:47,538 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:47,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:47,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:47,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:47,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:47,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:47,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:47,539 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:47,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:47,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:47,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:47,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:47,541 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:47,541 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:47,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:47,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:47,541 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:47,541 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:57:47,541 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:57:47,541 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:57:47,541 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:47,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:47,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:47,541 DEBUG [Thread-2783] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:57:47,541 DEBUG [Thread-2783] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:57:47,541 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:47,541 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:47,541 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:47,541 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:47,541 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:47,541 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:47,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(9158): Flush requested on 8dbd4df2f9769e1bcf08d64a8aceb9ad 2023-03-21 11:57:47,575 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8dbd4df2f9769e1bcf08d64a8aceb9ad 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-21 11:57:47,576 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dbd4df2f9769e1bcf08d64a8aceb9ad, store=A 2023-03-21 11:57:47,576 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:57:47,576 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dbd4df2f9769e1bcf08d64a8aceb9ad, store=B 2023-03-21 11:57:47,576 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:57:47,576 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dbd4df2f9769e1bcf08d64a8aceb9ad, store=C 2023-03-21 11:57:47,576 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:57:47,641 DEBUG [Thread-2783] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:47,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:47,643 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:57:47,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:57:47,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:57:47,644 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:57:47,644 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:47,644 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:57:47,644 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:57:47,644 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:47,645 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:47,645 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:47,645 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:47,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:47,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:47,645 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:47,645 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:57:47,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:47,646 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:47,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:57:47,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:47,646 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:57:47,646 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:57:47,646 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:57:47,646 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:57:47,646 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:47,646 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:57:47,646 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:57:47,646 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:57:47,646 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:47,647 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. started... 2023-03-21 11:57:47,647 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. as already flushing 2023-03-21 11:57:47,647 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:47,647 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:57:47,647 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:57:47,647 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:47,647 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:57:47,647 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:57:47,648 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:47,648 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:47,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:47,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:47,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:47,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:47,648 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:47,648 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:57:47,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:47,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:47,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:47,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:47,650 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:57:47,650 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3eb13c38[Count = 0] remaining members to acquire global barrier 2023-03-21 11:57:47,650 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:57:47,650 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:47,651 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:47,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:47,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:47,651 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:57:47,651 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:47,651 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:57:47,651 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:57:47,651 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:57:47,651 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:47,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:47,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:47,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:47,652 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:57:47,652 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:47,652 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:57:47,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:47,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:47,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:47,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:47,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:47,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:47,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:47,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:57:47,653 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:57:47,653 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:57:47,653 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:57:47,654 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:57:47,654 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:57:47,654 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:57:47,655 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:47,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:47,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:47,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:47,655 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:47,655 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:47,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:47,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:47,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:47,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:47,656 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:47,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:47,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:47,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:47,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:47,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:47,657 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:47,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:47,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:47,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:47,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:47,669 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:47,669 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:47,670 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:57:47,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:47,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:47,670 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:57:47,670 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:47,670 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:57:47,670 DEBUG [Thread-2783] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:57:47,670 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:47,670 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:47,670 DEBUG [Thread-2783] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:57:47,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:47,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:47,670 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:47,670 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:47,670 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:47,670 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:47,670 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:47,691 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:47,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48362 deadline: 1679399927687, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:47,770 DEBUG [Thread-2783] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:47,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:47,772 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:57:47,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:57:47,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:57:47,773 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:57:47,773 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:47,773 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:57:47,773 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:57:47,773 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:47,773 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:47,775 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:47,775 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:47,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:47,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:47,775 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:47,775 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:57:47,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:47,775 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:47,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:57:47,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:47,776 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:57:47,776 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:57:47,776 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:57:47,776 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:57:47,776 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:47,776 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:57:47,776 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:57:47,776 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:47,776 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. started... 2023-03-21 11:57:47,776 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:57:47,776 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. as already flushing 2023-03-21 11:57:47,776 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:47,776 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:57:47,776 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:57:47,776 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:47,776 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:57:47,776 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:57:47,777 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:47,777 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:47,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:47,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:47,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:47,777 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:47,777 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:57:47,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:47,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:47,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:47,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:47,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:47,778 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:57:47,778 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@61ae1faf[Count = 0] remaining members to acquire global barrier 2023-03-21 11:57:47,778 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:57:47,778 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:47,779 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:47,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:47,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:47,779 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:57:47,779 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:57:47,779 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:57:47,779 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:47,779 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:57:47,780 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:47,780 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:57:47,780 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:47,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:47,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:47,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:47,780 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:57:47,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:47,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:47,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:47,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:47,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:47,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:47,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:47,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:57:47,782 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:57:47,782 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:57:47,782 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:57:47,782 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:57:47,782 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:57:47,782 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:57:47,783 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:47,783 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:47,783 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:47,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:47,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:47,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:47,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:47,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:47,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:47,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:47,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:47,783 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:47,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:47,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:47,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:47,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:47,784 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:47,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:47,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:47,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:47,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:47,785 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:47,785 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:47,785 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:47,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:47,785 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:57:47,785 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:57:47,785 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:47,785 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:47,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:47,786 DEBUG [Thread-2783] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:57:47,785 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:47,786 DEBUG [Thread-2783] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:57:47,786 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:57:47,786 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:47,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:47,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:47,786 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:47,786 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:47,786 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:47,799 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:47,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48362 deadline: 1679399927792, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:47,886 DEBUG [Thread-2783] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:47,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:47,888 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:57:47,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:57:47,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:57:47,888 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:57:47,889 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:47,889 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:57:47,889 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:57:47,889 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:47,889 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:47,889 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:47,889 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:47,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:47,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:47,890 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:47,890 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:57:47,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:47,890 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:47,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:57:47,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:47,890 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:57:47,890 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:57:47,890 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:57:47,890 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:57:47,890 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:47,890 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:57:47,891 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:57:47,891 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:57:47,891 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:47,891 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. started... 2023-03-21 11:57:47,891 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. as already flushing 2023-03-21 11:57:47,891 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:47,891 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:57:47,891 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:57:47,891 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:47,891 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:57:47,891 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:57:47,892 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:47,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:47,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:47,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:47,892 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:47,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:47,892 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:47,892 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:57:47,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:47,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:47,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:47,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:47,893 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:57:47,893 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:57:47,893 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e68512c[Count = 0] remaining members to acquire global barrier 2023-03-21 11:57:47,893 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:47,893 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:47,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:47,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:47,894 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:57:47,894 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:47,894 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:57:47,894 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:57:47,894 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:57:47,894 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:47,894 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:57:47,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:47,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:47,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:47,894 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:47,895 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:57:47,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:47,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:47,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:47,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:47,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:47,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:47,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:47,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:57:47,896 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:57:47,896 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:57:47,896 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:57:47,896 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:57:47,896 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:57:47,896 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:57:47,897 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:47,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:47,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:47,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:47,897 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:47,897 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:47,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:47,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:47,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:47,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:47,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:47,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:47,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:47,899 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:47,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:47,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:47,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:47,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:47,900 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:47,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:47,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:47,901 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:47,901 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:47,901 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:47,901 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:47,902 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:47,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:47,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:47,901 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:57:47,902 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:57:47,901 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:47,902 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:47,902 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:47,902 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:47,902 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:47,902 DEBUG [Thread-2783] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:57:47,902 DEBUG [Thread-2783] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:57:47,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:47,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:47,981 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=156 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/A/67d31c5bbffc4ef98bec224823310122 2023-03-21 11:57:47,988 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=156 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/B/3913b29f536045c09825e00e42f26792 2023-03-21 11:57:47,996 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=156 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/C/86044c66c16e4ac78a4fad9ec58c6d99 2023-03-21 11:57:47,999 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/A/67d31c5bbffc4ef98bec224823310122 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/67d31c5bbffc4ef98bec224823310122 2023-03-21 11:57:48,002 DEBUG [Thread-2783] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:48,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:48,003 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/67d31c5bbffc4ef98bec224823310122, entries=200, sequenceid=156, filesize=14.1 K 2023-03-21 11:57:48,003 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/B/3913b29f536045c09825e00e42f26792 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/3913b29f536045c09825e00e42f26792 2023-03-21 11:57:48,004 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:57:48,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:57:48,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:57:48,005 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:57:48,005 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:48,005 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:57:48,005 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:57:48,005 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:48,005 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:48,006 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:48,006 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:48,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:48,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:48,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:48,006 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:48,006 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:57:48,007 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:48,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:57:48,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:48,007 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:57:48,007 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:57:48,007 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:57:48,007 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:57:48,007 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:48,007 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:57:48,007 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/3913b29f536045c09825e00e42f26792, entries=150, sequenceid=156, filesize=11.8 K 2023-03-21 11:57:48,007 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:48,007 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:57:48,007 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:57:48,007 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. started... 2023-03-21 11:57:48,008 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. as already flushing 2023-03-21 11:57:48,008 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:48,008 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:57:48,008 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:57:48,008 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:48,008 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:57:48,008 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:57:48,008 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/C/86044c66c16e4ac78a4fad9ec58c6d99 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/86044c66c16e4ac78a4fad9ec58c6d99 2023-03-21 11:57:48,008 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:48,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48362 deadline: 1679399928000, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:48,011 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/86044c66c16e4ac78a4fad9ec58c6d99, entries=150, sequenceid=156, filesize=11.8 K 2023-03-21 11:57:48,012 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 8dbd4df2f9769e1bcf08d64a8aceb9ad in 437ms, sequenceid=156, compaction requested=true 2023-03-21 11:57:48,012 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8dbd4df2f9769e1bcf08d64a8aceb9ad: 2023-03-21 11:57:48,012 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-21 11:57:48,012 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 9 store files, 0 compacting, 9 eligible, 16 blocking 2023-03-21 11:57:48,012 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 9 store files, 0 compacting, 9 eligible, 16 blocking 2023-03-21 11:57:48,012 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-21 11:57:48,012 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-21 11:57:48,014 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 9 files of size 109614 starting at candidate #0 after considering 28 permutations with 28 in ratio 2023-03-21 11:57:48,014 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 9 files of size 107224 starting at candidate #0 after considering 28 permutations with 28 in ratio 2023-03-21 11:57:48,014 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1912): 8dbd4df2f9769e1bcf08d64a8aceb9ad/A is initiating minor compaction (all files) 2023-03-21 11:57:48,014 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1912): 8dbd4df2f9769e1bcf08d64a8aceb9ad/B is initiating minor compaction (all files) 2023-03-21 11:57:48,014 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8dbd4df2f9769e1bcf08d64a8aceb9ad/A in TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:48,014 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8dbd4df2f9769e1bcf08d64a8aceb9ad/B in TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:48,014 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/efdec08f12ce4e268b133e793464c267, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/52168d3b40b74cde9f216a960c66fab0, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/1848cf8a974748bcbfe3f9f7e952b709, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/ee565f7fbdee45298a18bef5464d4206, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/dc37a1f2797d499ea1d6e8aa8a1367e4, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/a53e2fa6af314745818d764952284452, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/0e62ed6012624fa889f1cbd819384503, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/da5e172d03fd4ff2822262d9d22d3ea4, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/3913b29f536045c09825e00e42f26792] into tmpdir=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp, totalSize=104.7 K 2023-03-21 11:57:48,014 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/48f70e5740e24bf2b2ce7bf8b98eea79, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/7a7ab837338a4b68bac517fbb992affa, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/46401f06a61b4f9c9189fc6dacf34687, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/0e54f1c8c20443339da361430aebe1fa, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/bb0436b397b24ef58e44d776c40f73bc, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/d18ce65c0770425b9f9bf47b5f20fe8a, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/118e5d59e8a04a8bb6610dae3e71dc70, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/93fb6dc50f6543aab2b0b2cb1426906b, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/67d31c5bbffc4ef98bec224823310122] into tmpdir=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp, totalSize=107.0 K 2023-03-21 11:57:48,014 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting efdec08f12ce4e268b133e793464c267, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1679399859652 2023-03-21 11:57:48,014 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 48f70e5740e24bf2b2ce7bf8b98eea79, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1679399859652 2023-03-21 11:57:48,014 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 52168d3b40b74cde9f216a960c66fab0, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=33, earliestPutTs=1679399859675 2023-03-21 11:57:48,014 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 7a7ab837338a4b68bac517fbb992affa, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=33, earliestPutTs=1679399859675 2023-03-21 11:57:48,015 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 1848cf8a974748bcbfe3f9f7e952b709, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=45, earliestPutTs=1679399860299 2023-03-21 11:57:48,015 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 46401f06a61b4f9c9189fc6dacf34687, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=45, earliestPutTs=1679399860299 2023-03-21 11:57:48,015 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting ee565f7fbdee45298a18bef5464d4206, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=79, earliestPutTs=1679399861456 2023-03-21 11:57:48,015 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 0e54f1c8c20443339da361430aebe1fa, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=79, earliestPutTs=1679399861456 2023-03-21 11:57:48,015 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting dc37a1f2797d499ea1d6e8aa8a1367e4, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=91, earliestPutTs=1679399862597 2023-03-21 11:57:48,015 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting bb0436b397b24ef58e44d776c40f73bc, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=91, earliestPutTs=1679399862597 2023-03-21 11:57:48,015 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting a53e2fa6af314745818d764952284452, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=111, earliestPutTs=1679399863660 2023-03-21 11:57:48,015 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting d18ce65c0770425b9f9bf47b5f20fe8a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=111, earliestPutTs=1679399863660 2023-03-21 11:57:48,016 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 0e62ed6012624fa889f1cbd819384503, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=123, earliestPutTs=1679399864010 2023-03-21 11:57:48,016 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 118e5d59e8a04a8bb6610dae3e71dc70, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=123, earliestPutTs=1679399864010 2023-03-21 11:57:48,016 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting da5e172d03fd4ff2822262d9d22d3ea4, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=143, earliestPutTs=1679399865173 2023-03-21 11:57:48,016 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 93fb6dc50f6543aab2b0b2cb1426906b, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=143, earliestPutTs=1679399865173 2023-03-21 11:57:48,016 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 3913b29f536045c09825e00e42f26792, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=156, earliestPutTs=1679399866433 2023-03-21 11:57:48,016 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 67d31c5bbffc4ef98bec224823310122, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=156, earliestPutTs=1679399866402 2023-03-21 11:57:48,022 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:48,022 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:48,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:48,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:48,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:48,022 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:48,022 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:57:48,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:48,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:48,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:48,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:48,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:48,024 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:57:48,024 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@527cd29a[Count = 0] remaining members to acquire global barrier 2023-03-21 11:57:48,024 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:57:48,024 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:48,025 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:48,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:48,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:48,025 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:57:48,025 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:57:48,025 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:48,025 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:57:48,025 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:57:48,026 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:57:48,026 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:48,026 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:48,026 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:57:48,027 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8dbd4df2f9769e1bcf08d64a8aceb9ad#A#compaction#372 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 57.69 MB/second 2023-03-21 11:57:48,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:48,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:48,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:48,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:48,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:48,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:48,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:48,028 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] throttle.PressureAwareThroughputController(145): 8dbd4df2f9769e1bcf08d64a8aceb9ad#B#compaction#373 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 57.69 MB/second 2023-03-21 11:57:48,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:48,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:48,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:48,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:57:48,029 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:57:48,029 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:57:48,029 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:57:48,029 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:57:48,029 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:57:48,029 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:57:48,032 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:48,032 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:48,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:48,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:48,032 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:48,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:48,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:48,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:48,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:48,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:48,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:48,032 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:48,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:48,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:48,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:48,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:48,033 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:48,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:48,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:48,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:48,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:48,035 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:48,035 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:48,035 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:48,035 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:48,035 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:48,035 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:48,035 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:48,035 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:57:48,035 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:57:48,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:48,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:48,035 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:48,035 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:48,035 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:57:48,035 DEBUG [Thread-2783] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:57:48,035 DEBUG [Thread-2783] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:57:48,035 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:48,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:48,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:48,136 DEBUG [Thread-2783] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:48,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:48,137 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:57:48,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:57:48,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:57:48,138 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:57:48,138 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:48,138 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:57:48,138 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:57:48,139 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:48,139 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:48,144 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:48,144 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:48,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:48,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:48,145 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:48,145 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:57:48,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:48,145 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:48,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:57:48,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:48,145 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:57:48,146 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:57:48,146 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:57:48,146 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:57:48,146 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:48,146 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:57:48,146 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:57:48,146 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:57:48,146 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:48,146 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. started... 2023-03-21 11:57:48,146 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8dbd4df2f9769e1bcf08d64a8aceb9ad 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-21 11:57:48,146 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dbd4df2f9769e1bcf08d64a8aceb9ad, store=A 2023-03-21 11:57:48,146 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:57:48,146 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dbd4df2f9769e1bcf08d64a8aceb9ad, store=B 2023-03-21 11:57:48,146 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:57:48,146 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dbd4df2f9769e1bcf08d64a8aceb9ad, store=C 2023-03-21 11:57:48,146 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:57:48,152 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=175 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/A/dceaf26e29fb4b62961d135a5f226aed 2023-03-21 11:57:48,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(9158): Flush requested on 8dbd4df2f9769e1bcf08d64a8aceb9ad 2023-03-21 11:57:48,318 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. as already flushing 2023-03-21 11:57:48,389 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:48,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48362 deadline: 1679399928380, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:48,448 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/B/349ea59266304913a63e4d1e0a769cd6 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/349ea59266304913a63e4d1e0a769cd6 2023-03-21 11:57:48,448 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/A/906ec9a526024ace9409a51f7d3dbc5c as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/906ec9a526024ace9409a51f7d3dbc5c 2023-03-21 11:57:48,451 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 9 (all) file(s) in 8dbd4df2f9769e1bcf08d64a8aceb9ad/A of 8dbd4df2f9769e1bcf08d64a8aceb9ad into 906ec9a526024ace9409a51f7d3dbc5c(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-21 11:57:48,451 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1652): Completed compaction of 9 (all) file(s) in 8dbd4df2f9769e1bcf08d64a8aceb9ad/B of 8dbd4df2f9769e1bcf08d64a8aceb9ad into 349ea59266304913a63e4d1e0a769cd6(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-21 11:57:48,451 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8dbd4df2f9769e1bcf08d64a8aceb9ad: 2023-03-21 11:57:48,452 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8dbd4df2f9769e1bcf08d64a8aceb9ad: 2023-03-21 11:57:48,452 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad., storeName=8dbd4df2f9769e1bcf08d64a8aceb9ad/A, priority=7, startTime=1679399868012; duration=0sec 2023-03-21 11:57:48,452 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad., storeName=8dbd4df2f9769e1bcf08d64a8aceb9ad/B, priority=7, startTime=1679399868012; duration=0sec 2023-03-21 11:57:48,452 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-21 11:57:48,452 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-21 11:57:48,452 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 9 store files, 0 compacting, 9 eligible, 16 blocking 2023-03-21 11:57:48,453 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 9 files of size 107224 starting at candidate #0 after considering 28 permutations with 28 in ratio 2023-03-21 11:57:48,453 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1912): 8dbd4df2f9769e1bcf08d64a8aceb9ad/C is initiating minor compaction (all files) 2023-03-21 11:57:48,453 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8dbd4df2f9769e1bcf08d64a8aceb9ad/C in TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:48,454 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/3ad32817e68d4feda55ff068caa3304d, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/ba5820a8f0f045988e1819b794743b47, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/c2558cdf5d6b4f219c5d7890dc754e3f, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/c59afd9df3de4dad85f72d1caa39551f, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/f43be7b7323c479c98d2ce6f2254d49d, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/06cdbe75ba0642bcac8ccdaa2114def6, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/f16c7a152162417982b660dc6ae0f1c5, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/b962320e8ae24525bed2262267651f53, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/86044c66c16e4ac78a4fad9ec58c6d99] into tmpdir=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp, totalSize=104.7 K 2023-03-21 11:57:48,454 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 3ad32817e68d4feda55ff068caa3304d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1679399859652 2023-03-21 11:57:48,454 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting ba5820a8f0f045988e1819b794743b47, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=33, earliestPutTs=1679399859675 2023-03-21 11:57:48,454 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting c2558cdf5d6b4f219c5d7890dc754e3f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=45, earliestPutTs=1679399860299 2023-03-21 11:57:48,455 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting c59afd9df3de4dad85f72d1caa39551f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=79, earliestPutTs=1679399861456 2023-03-21 11:57:48,455 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting f43be7b7323c479c98d2ce6f2254d49d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=91, earliestPutTs=1679399862597 2023-03-21 11:57:48,455 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 06cdbe75ba0642bcac8ccdaa2114def6, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=111, earliestPutTs=1679399863660 2023-03-21 11:57:48,455 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting f16c7a152162417982b660dc6ae0f1c5, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=123, earliestPutTs=1679399864010 2023-03-21 11:57:48,455 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting b962320e8ae24525bed2262267651f53, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=143, earliestPutTs=1679399865173 2023-03-21 11:57:48,456 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 86044c66c16e4ac78a4fad9ec58c6d99, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=156, earliestPutTs=1679399866433 2023-03-21 11:57:48,469 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8dbd4df2f9769e1bcf08d64a8aceb9ad#C#compaction#376 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 57.69 MB/second 2023-03-21 11:57:48,497 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:48,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48362 deadline: 1679399928490, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:48,561 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=175 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/B/3a398d0612d04e3bb07f16f19e8a52cf 2023-03-21 11:57:48,702 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:48,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48362 deadline: 1679399928698, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:48,876 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/C/b51026cea1e14b69a357d1f2a48cfbd0 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/b51026cea1e14b69a357d1f2a48cfbd0 2023-03-21 11:57:48,879 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 9 (all) file(s) in 8dbd4df2f9769e1bcf08d64a8aceb9ad/C of 8dbd4df2f9769e1bcf08d64a8aceb9ad into b51026cea1e14b69a357d1f2a48cfbd0(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-21 11:57:48,879 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8dbd4df2f9769e1bcf08d64a8aceb9ad: 2023-03-21 11:57:48,880 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad., storeName=8dbd4df2f9769e1bcf08d64a8aceb9ad/C, priority=7, startTime=1679399868012; duration=0sec 2023-03-21 11:57:48,880 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-21 11:57:48,969 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=175 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/C/b35cf28e8e7146b6899cfa8fc5fe48ec 2023-03-21 11:57:48,972 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/A/dceaf26e29fb4b62961d135a5f226aed as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/dceaf26e29fb4b62961d135a5f226aed 2023-03-21 11:57:48,975 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/dceaf26e29fb4b62961d135a5f226aed, entries=150, sequenceid=175, filesize=11.8 K 2023-03-21 11:57:48,976 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/B/3a398d0612d04e3bb07f16f19e8a52cf as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/3a398d0612d04e3bb07f16f19e8a52cf 2023-03-21 11:57:48,979 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/3a398d0612d04e3bb07f16f19e8a52cf, entries=150, sequenceid=175, filesize=11.8 K 2023-03-21 11:57:48,979 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/C/b35cf28e8e7146b6899cfa8fc5fe48ec as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/b35cf28e8e7146b6899cfa8fc5fe48ec 2023-03-21 11:57:48,982 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/b35cf28e8e7146b6899cfa8fc5fe48ec, entries=150, sequenceid=175, filesize=11.8 K 2023-03-21 11:57:48,983 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 8dbd4df2f9769e1bcf08d64a8aceb9ad in 837ms, sequenceid=175, compaction requested=false 2023-03-21 11:57:48,983 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8dbd4df2f9769e1bcf08d64a8aceb9ad: 2023-03-21 11:57:48,983 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:48,983 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:57:48,983 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:57:48,983 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:48,983 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:57:48,983 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:57:48,984 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:48,984 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:48,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:48,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:48,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:48,984 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:48,984 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:57:48,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:48,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:48,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:48,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:48,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:48,985 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:57:48,985 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@651b6411[Count = 0] remaining members to acquire global barrier 2023-03-21 11:57:48,985 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:57:48,985 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:48,986 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:48,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:48,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:48,986 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:57:48,986 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:57:48,986 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:57:48,986 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:48,986 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:57:48,986 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:57:48,987 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:48,986 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:48,987 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:57:48,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:48,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:48,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:48,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:48,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:48,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:48,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:48,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:48,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:48,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:48,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:57:48,988 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:57:48,988 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:57:48,988 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:57:48,989 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:57:48,989 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:57:48,989 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:57:48,989 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:48,989 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:48,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:48,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:48,989 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:48,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:48,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:48,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:48,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:48,990 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:48,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:48,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:48,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:48,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:48,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:48,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:48,991 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:48,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:48,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:48,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:48,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:48,992 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:48,992 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:48,992 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:48,992 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:48,992 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:57:48,992 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:57:48,992 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:48,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:48,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:48,992 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:48,992 DEBUG [Thread-2783] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:57:48,992 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:57:48,992 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:48,992 DEBUG [Thread-2783] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:57:48,992 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:48,992 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:48,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:48,992 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:48,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:49,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(9158): Flush requested on 8dbd4df2f9769e1bcf08d64a8aceb9ad 2023-03-21 11:57:49,009 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8dbd4df2f9769e1bcf08d64a8aceb9ad 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-03-21 11:57:49,009 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dbd4df2f9769e1bcf08d64a8aceb9ad, store=A 2023-03-21 11:57:49,009 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:57:49,009 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dbd4df2f9769e1bcf08d64a8aceb9ad, store=B 2023-03-21 11:57:49,009 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:57:49,009 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dbd4df2f9769e1bcf08d64a8aceb9ad, store=C 2023-03-21 11:57:49,009 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:57:49,087 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:49,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48362 deadline: 1679399929079, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,092 DEBUG [Thread-2783] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:49,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:49,094 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:57:49,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:57:49,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:57:49,095 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:57:49,095 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:49,095 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:57:49,095 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:57:49,095 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:49,095 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:49,096 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,096 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:49,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:49,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:49,096 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,096 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:57:49,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:49,096 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:49,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:57:49,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:49,096 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:57:49,097 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:57:49,097 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:57:49,097 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:57:49,097 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:49,097 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:57:49,097 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:57:49,097 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:57:49,097 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:49,097 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. started... 2023-03-21 11:57:49,097 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. as already flushing 2023-03-21 11:57:49,097 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:49,097 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:57:49,097 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:57:49,097 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,097 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:57:49,097 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:57:49,098 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:49,098 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:49,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:49,098 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:49,098 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:57:49,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:49,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:49,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:49,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:49,099 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:57:49,099 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74594c7a[Count = 0] remaining members to acquire global barrier 2023-03-21 11:57:49,099 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:57:49,099 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:49,100 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:49,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:49,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:49,100 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,100 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:57:49,100 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:57:49,100 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:57:49,100 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:57:49,100 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:57:49,100 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:49,100 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,100 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:57:49,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:49,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:49,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:49,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:49,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:49,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:49,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:49,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:57:49,102 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:57:49,102 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:57:49,102 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:57:49,102 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:57:49,102 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:57:49,102 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:57:49,104 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:49,104 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:49,104 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:49,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:49,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:49,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:49,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:49,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:49,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:49,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:49,105 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:49,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:49,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:49,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:49,106 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:49,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:49,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:49,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,107 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:49,107 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,107 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:57:49,107 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:57:49,107 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:49,107 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:49,107 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:57:49,107 DEBUG [Thread-2783] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:57:49,107 DEBUG [Thread-2783] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:57:49,107 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:49,108 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:49,108 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,108 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:49,108 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:49,108 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:49,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:49,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:49,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:49,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:49,191 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:49,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48362 deadline: 1679399929188, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,207 DEBUG [Thread-2783] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:49,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:49,210 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:57:49,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:57:49,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:57:49,211 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:57:49,211 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:49,211 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:57:49,211 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:57:49,211 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:49,211 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:49,212 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,212 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:49,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:49,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:49,212 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,212 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:57:49,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:49,212 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:49,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:57:49,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:49,213 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:57:49,213 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:57:49,213 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:57:49,213 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:57:49,213 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:49,213 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:57:49,213 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:57:49,213 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:57:49,213 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:49,213 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. started... 2023-03-21 11:57:49,213 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. as already flushing 2023-03-21 11:57:49,213 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:49,213 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:57:49,213 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:57:49,213 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,213 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:57:49,213 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:57:49,214 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,214 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:49,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:49,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:49,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:49,215 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:49,215 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:57:49,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:49,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:49,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:49,215 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:57:49,215 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@38f482ec[Count = 0] remaining members to acquire global barrier 2023-03-21 11:57:49,215 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:57:49,215 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:49,216 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:49,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:49,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:49,216 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:57:49,216 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:57:49,216 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,216 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:57:49,216 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:57:49,217 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,217 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:57:49,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:49,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:49,217 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:49,217 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:57:49,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:49,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:49,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:49,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:49,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:49,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:57:49,218 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:57:49,218 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:57:49,219 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:57:49,219 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:57:49,219 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:57:49,219 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:57:49,220 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:49,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:49,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:49,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:49,220 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:49,221 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:49,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:49,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:49,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:49,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:49,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:49,221 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:49,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:49,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:49,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:49,222 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:49,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:49,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,224 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:49,224 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:49,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:49,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:49,224 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,224 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:57:49,224 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:49,224 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:57:49,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:49,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:49,224 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:49,224 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:57:49,224 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:49,224 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,224 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:49,224 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:49,224 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:49,224 DEBUG [Thread-2783] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:57:49,224 DEBUG [Thread-2783] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:57:49,324 DEBUG [Thread-2783] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:49,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:49,326 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:57:49,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:57:49,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:57:49,327 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:57:49,327 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:49,327 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:57:49,327 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:57:49,327 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:49,327 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:49,328 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:49,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:49,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:49,328 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:49,328 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,328 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:57:49,329 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:49,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:57:49,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:49,329 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:57:49,329 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:57:49,329 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:57:49,329 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:57:49,329 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:49,329 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:57:49,329 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:57:49,329 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:57:49,329 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:49,329 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. started... 2023-03-21 11:57:49,329 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. as already flushing 2023-03-21 11:57:49,329 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:49,329 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:57:49,329 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:57:49,329 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,330 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:57:49,330 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:57:49,330 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,330 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:49,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:49,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:49,330 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:49,331 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:57:49,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:49,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:49,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:49,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:49,332 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:57:49,332 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@c287f6a[Count = 0] remaining members to acquire global barrier 2023-03-21 11:57:49,332 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:57:49,332 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:49,332 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:49,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:49,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:49,332 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:57:49,332 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:57:49,332 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:57:49,332 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,332 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:57:49,333 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,333 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:57:49,333 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:49,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:49,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:49,333 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:57:49,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:49,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:49,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:49,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:49,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:49,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:57:49,335 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:57:49,335 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:57:49,335 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:57:49,335 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:57:49,335 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:57:49,335 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:57:49,337 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:49,337 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:49,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:49,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:49,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:49,337 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:49,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:49,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:49,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:49,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:49,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:49,337 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:49,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:49,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:49,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:49,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,338 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:49,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:49,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,339 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,339 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:49,339 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:49,339 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:49,339 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:49,339 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:57:49,339 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:57:49,339 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:57:49,339 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:49,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:49,340 DEBUG [Thread-2783] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:57:49,340 DEBUG [Thread-2783] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:57:49,339 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:49,339 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:49,340 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:49,340 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:49,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:49,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:49,395 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:49,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48362 deadline: 1679399929392, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,415 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=192 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/A/2abf421417db453f89a497047c79f2f2 2023-03-21 11:57:49,440 DEBUG [Thread-2783] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:49,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:49,441 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:57:49,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:57:49,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:57:49,442 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:57:49,442 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:49,442 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:57:49,442 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:57:49,443 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:49,443 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:49,443 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:49,443 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:49,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:49,444 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,444 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:57:49,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:49,444 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:49,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:57:49,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:49,444 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:57:49,445 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:57:49,445 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:57:49,445 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:57:49,445 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:49,445 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:57:49,445 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:57:49,445 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:57:49,445 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:49,445 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. started... 2023-03-21 11:57:49,445 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. as already flushing 2023-03-21 11:57:49,445 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:49,445 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:57:49,445 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:57:49,445 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,445 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:57:49,445 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:57:49,446 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,446 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:49,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:49,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:49,446 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:49,446 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:57:49,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:49,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:49,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:49,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:49,447 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:57:49,447 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@322db912[Count = 0] remaining members to acquire global barrier 2023-03-21 11:57:49,447 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:57:49,447 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:49,448 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:49,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:49,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:49,448 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,448 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:57:49,448 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:57:49,448 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:57:49,448 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:57:49,448 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,448 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:57:49,449 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:49,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:49,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:49,449 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:57:49,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:49,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:49,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:49,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:49,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:49,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:57:49,450 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:57:49,450 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:57:49,450 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:57:49,450 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:57:49,450 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:57:49,450 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:57:49,451 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:49,451 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:49,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:49,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:49,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:49,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:49,451 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:49,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:49,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:49,451 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:49,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:49,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:49,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:49,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:49,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:49,452 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:49,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:49,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,454 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,454 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:49,454 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:49,454 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:57:49,454 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:57:49,454 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:57:49,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:49,454 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:49,454 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:49,454 DEBUG [Thread-2783] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:57:49,454 DEBUG [Thread-2783] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:57:49,454 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:49,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:49,454 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,454 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:49,454 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:49,454 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:49,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:49,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:49,554 DEBUG [Thread-2783] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:49,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:49,556 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:57:49,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:57:49,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:57:49,557 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:57:49,557 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:49,557 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:57:49,557 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:57:49,557 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:49,557 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:49,558 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,558 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:49,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:49,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:49,558 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,558 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:57:49,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:49,559 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:49,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:57:49,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:49,559 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:57:49,559 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:57:49,559 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:57:49,559 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:57:49,559 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:49,559 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:57:49,559 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:57:49,559 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:57:49,559 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:49,559 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. started... 2023-03-21 11:57:49,559 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. as already flushing 2023-03-21 11:57:49,559 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:49,559 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:57:49,560 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:57:49,560 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,560 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:57:49,560 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:57:49,560 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,560 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:49,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:49,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:49,560 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:49,561 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:57:49,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:49,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:49,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:49,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:49,561 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:57:49,561 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@c2909b1[Count = 0] remaining members to acquire global barrier 2023-03-21 11:57:49,561 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:57:49,561 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:49,562 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:49,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:49,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:49,562 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:57:49,562 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,562 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:57:49,562 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:57:49,562 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:57:49,563 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:57:49,563 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,563 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:49,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:49,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:49,563 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:57:49,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:49,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:49,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:49,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:49,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:49,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:57:49,564 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:57:49,564 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:57:49,564 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:57:49,564 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:57:49,564 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:57:49,564 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:57:49,565 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:49,565 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:49,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:49,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:49,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:49,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:49,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:49,565 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:49,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:49,565 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:49,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:49,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:49,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:49,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:49,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:49,566 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:49,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:49,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,567 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,567 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:49,567 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:49,567 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:49,567 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:57:49,567 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:49,567 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:57:49,567 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:57:49,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:49,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:49,567 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:49,568 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,568 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:49,568 DEBUG [Thread-2783] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:57:49,568 DEBUG [Thread-2783] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:57:49,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:49,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:49,568 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:49,568 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:49,668 DEBUG [Thread-2783] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:49,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:49,669 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:57:49,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:57:49,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:57:49,670 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:57:49,670 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:49,670 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:57:49,671 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:57:49,671 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:49,671 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:49,672 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,672 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:49,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:49,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:49,672 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,672 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:57:49,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:49,672 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:49,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:57:49,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:49,673 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:57:49,673 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:57:49,673 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:57:49,673 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:57:49,673 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:49,673 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:57:49,673 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:57:49,673 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:57:49,673 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:49,673 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. started... 2023-03-21 11:57:49,673 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. as already flushing 2023-03-21 11:57:49,673 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:49,673 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:57:49,673 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:57:49,673 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,674 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:57:49,674 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:57:49,674 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:49,674 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:49,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:49,674 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:49,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:49,675 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:57:49,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:49,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:49,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:49,676 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:57:49,676 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@10d6193a[Count = 0] remaining members to acquire global barrier 2023-03-21 11:57:49,676 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:57:49,676 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:49,677 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:49,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:49,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:49,677 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:57:49,677 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,677 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:57:49,677 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:57:49,677 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:57:49,678 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:57:49,678 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,678 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:49,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:49,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:49,678 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:57:49,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:49,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:49,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:49,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:49,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:49,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:57:49,680 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:57:49,680 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:57:49,680 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:57:49,680 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:57:49,680 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:57:49,680 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:57:49,680 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:49,680 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:49,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:49,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:49,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:49,681 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:49,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:49,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:49,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:49,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:49,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:49,681 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:49,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:49,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:49,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:49,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,682 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:49,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:49,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,683 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,683 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:49,684 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:49,684 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:49,684 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,684 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:57:49,684 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:57:49,684 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:49,684 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:49,684 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:49,684 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:49,684 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:49,684 DEBUG [Thread-2783] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:57:49,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:49,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:49,684 DEBUG [Thread-2783] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:57:49,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:49,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:49,701 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:49,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48362 deadline: 1679399929697, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,784 DEBUG [Thread-2783] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:49,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:49,786 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:57:49,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:57:49,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:57:49,787 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:57:49,787 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:49,787 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:57:49,787 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:57:49,788 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:49,788 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:49,788 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:49,788 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:49,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:49,789 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,789 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:57:49,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:49,789 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:49,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:57:49,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:49,789 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:57:49,789 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:57:49,789 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:57:49,790 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:57:49,790 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:49,790 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:57:49,790 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:57:49,790 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:57:49,790 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:49,790 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. started... 2023-03-21 11:57:49,790 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. as already flushing 2023-03-21 11:57:49,790 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:49,790 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:57:49,790 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:57:49,790 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,790 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:57:49,790 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:57:49,791 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,791 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:49,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:49,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:49,791 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:49,791 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:57:49,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:49,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:49,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:49,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:49,792 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:57:49,792 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5fcaf765[Count = 0] remaining members to acquire global barrier 2023-03-21 11:57:49,792 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:57:49,792 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:49,793 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,793 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:57:49,793 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:49,793 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:49,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:49,793 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:57:49,794 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:57:49,794 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:57:49,794 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:57:49,794 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:49,794 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:57:49,794 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:49,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:49,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:49,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:49,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:49,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:49,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:49,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:57:49,796 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:57:49,796 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:57:49,796 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:57:49,796 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:57:49,796 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:57:49,796 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:57:49,797 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:49,797 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:49,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:49,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:49,797 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:49,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:49,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:49,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:49,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:49,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:49,797 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:49,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:49,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:49,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:49,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:49,798 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:49,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:49,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,801 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,801 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:49,801 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:49,801 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:49,801 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:57:49,801 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:57:49,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:49,801 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:49,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:49,801 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:49,801 DEBUG [Thread-2783] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:57:49,801 DEBUG [Thread-2783] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:57:49,801 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,801 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:57:49,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:49,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:49,801 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:49,801 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:49,801 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:49,823 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=192 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/B/c307cf7252954120a77104dd171f9519 2023-03-21 11:57:49,901 DEBUG [Thread-2783] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:49,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:49,903 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:57:49,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:57:49,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:57:49,903 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:57:49,903 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:49,903 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:57:49,904 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:57:49,904 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:49,904 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:49,905 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:49,905 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:49,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:49,905 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,905 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:57:49,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:49,905 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:49,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:57:49,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:49,905 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:57:49,906 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:57:49,906 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:57:49,906 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:57:49,906 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:49,906 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:57:49,906 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:57:49,906 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:57:49,906 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:49,906 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. started... 2023-03-21 11:57:49,906 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. as already flushing 2023-03-21 11:57:49,906 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:49,906 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:57:49,906 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:57:49,906 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,906 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:57:49,906 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:57:49,907 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:49,907 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:49,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:49,907 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:49,907 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:57:49,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:49,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:49,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:49,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:49,908 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:57:49,908 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@f030a7[Count = 0] remaining members to acquire global barrier 2023-03-21 11:57:49,908 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:57:49,908 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:49,909 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:49,909 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:49,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:49,909 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:57:49,909 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:57:49,909 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,909 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:57:49,909 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:57:49,909 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:57:49,909 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:49,909 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,910 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:57:49,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:49,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:49,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:49,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:49,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:49,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:49,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:49,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:57:49,911 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:57:49,911 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:57:49,911 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:57:49,911 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:57:49,911 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:57:49,911 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:57:49,912 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:49,912 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:49,912 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:49,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:49,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:49,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:49,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:49,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:49,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:49,912 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:49,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:49,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:49,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:49,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:49,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:49,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,913 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:49,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:49,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,915 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:49,915 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:49,915 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:57:49,915 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:57:49,915 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:49,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:49,915 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:49,915 DEBUG [Thread-2783] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:57:49,915 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:49,915 DEBUG [Thread-2783] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:57:49,915 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:49,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:49,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:49,915 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:49,915 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:49,915 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:49,915 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:50,015 DEBUG [Thread-2783] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:50,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:50,017 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:57:50,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:57:50,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:57:50,018 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:57:50,018 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:50,018 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:57:50,018 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:57:50,018 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:50,018 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:50,019 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:50,019 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:50,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:50,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:50,019 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:50,019 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:57:50,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:50,019 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:50,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:57:50,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:50,020 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:57:50,020 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:57:50,020 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:57:50,020 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:57:50,020 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:50,020 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:57:50,020 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:57:50,020 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:57:50,020 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:50,020 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. started... 2023-03-21 11:57:50,020 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. as already flushing 2023-03-21 11:57:50,020 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:50,020 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:57:50,020 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:57:50,021 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:50,021 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:57:50,021 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:57:50,021 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:50,021 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:50,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:50,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:50,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:50,021 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:50,022 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:57:50,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:50,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:50,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:50,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:50,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:50,022 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:57:50,023 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4450c589[Count = 0] remaining members to acquire global barrier 2023-03-21 11:57:50,023 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:57:50,023 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:50,024 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:50,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:50,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:50,024 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:57:50,024 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:57:50,024 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:50,024 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:57:50,024 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:57:50,025 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:50,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:50,025 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:57:50,025 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:50,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:50,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:50,025 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:57:50,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:50,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:50,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:50,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:50,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:50,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:50,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:50,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:57:50,027 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:57:50,027 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:57:50,027 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:57:50,027 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:57:50,027 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:57:50,027 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:57:50,028 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:50,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:50,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:50,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:50,028 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:50,028 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:50,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:50,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:50,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:50,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:50,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:50,028 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:50,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:50,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:50,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:50,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:50,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:50,029 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:50,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:50,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:50,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:50,030 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:50,030 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:50,031 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:50,031 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:50,031 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:50,031 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:57:50,031 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:57:50,031 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:50,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:50,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:50,031 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:50,031 DEBUG [Thread-2783] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:57:50,031 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:50,031 DEBUG [Thread-2783] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:57:50,031 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:50,031 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:50,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:50,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:50,131 DEBUG [Thread-2783] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:50,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:50,133 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:57:50,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:57:50,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:57:50,134 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:57:50,134 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:50,134 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:57:50,134 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:57:50,134 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:50,134 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:50,135 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:50,135 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:50,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:50,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:50,135 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:50,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:50,135 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:57:50,135 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:50,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:57:50,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:50,136 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:57:50,136 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:57:50,136 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:57:50,136 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:57:50,136 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:50,136 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:57:50,136 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:57:50,136 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:57:50,136 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:50,136 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. started... 2023-03-21 11:57:50,136 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. as already flushing 2023-03-21 11:57:50,136 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:50,136 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:57:50,136 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:57:50,136 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:50,136 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:57:50,136 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:57:50,137 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:50,137 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:50,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:50,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:50,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:50,137 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:50,137 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:57:50,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:50,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:50,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:50,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:50,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:50,138 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:57:50,139 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@646f1bf0[Count = 0] remaining members to acquire global barrier 2023-03-21 11:57:50,139 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:57:50,139 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:50,139 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:50,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:50,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:50,139 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:57:50,139 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:57:50,139 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:57:50,139 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:50,139 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:57:50,140 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:50,140 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:57:50,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:50,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:50,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:50,140 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:50,140 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:57:50,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:50,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:50,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:50,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:50,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:50,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:50,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:50,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:57:50,142 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:57:50,142 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:57:50,142 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:57:50,142 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:57:50,142 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:57:50,142 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:57:50,143 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:50,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:50,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:50,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:50,143 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:50,144 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:50,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:50,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:50,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:50,144 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:50,145 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:50,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:50,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:50,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:50,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:50,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:50,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:50,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:50,146 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:50,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:50,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:50,147 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:57:50,147 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:57:50,147 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:50,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:50,146 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:50,146 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:50,147 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:50,147 DEBUG [Thread-2783] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:57:50,147 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:57:50,147 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:50,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:50,147 DEBUG [Thread-2783] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:57:50,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:50,147 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:50,147 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:50,147 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:50,147 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:50,204 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:50,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48362 deadline: 1679399930202, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:50,245 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=192 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/C/059653d6da58499db49c19ab54892bad 2023-03-21 11:57:50,247 DEBUG [Thread-2783] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:50,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:50,250 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:57:50,250 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/A/2abf421417db453f89a497047c79f2f2 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/2abf421417db453f89a497047c79f2f2 2023-03-21 11:57:50,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:57:50,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:57:50,255 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:57:50,255 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:50,255 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:57:50,255 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:57:50,256 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:50,256 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:50,256 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:50,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:50,256 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:50,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:50,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:50,257 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:50,257 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:57:50,257 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:50,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:57:50,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:50,257 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:57:50,257 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:57:50,257 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:57:50,257 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:57:50,258 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:50,258 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:57:50,258 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/2abf421417db453f89a497047c79f2f2, entries=200, sequenceid=192, filesize=14.1 K 2023-03-21 11:57:50,258 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:57:50,258 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:57:50,258 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:50,258 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. started... 2023-03-21 11:57:50,258 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. as already flushing 2023-03-21 11:57:50,258 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:50,258 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:57:50,258 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:57:50,258 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:50,258 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:57:50,258 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:57:50,258 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/B/c307cf7252954120a77104dd171f9519 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/c307cf7252954120a77104dd171f9519 2023-03-21 11:57:50,259 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:50,259 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:50,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:50,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:50,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:50,259 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:50,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:50,259 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:57:50,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:50,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:50,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:50,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:50,260 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:57:50,260 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c16b9c7[Count = 0] remaining members to acquire global barrier 2023-03-21 11:57:50,260 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:57:50,260 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:50,261 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:50,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:50,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:50,261 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:57:50,261 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:57:50,261 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:50,261 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:57:50,261 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:57:50,262 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:50,262 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:57:50,262 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:50,262 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:57:50,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:50,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:50,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:50,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:50,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:50,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:50,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:50,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:50,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:50,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:50,264 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/c307cf7252954120a77104dd171f9519, entries=150, sequenceid=192, filesize=11.8 K 2023-03-21 11:57:50,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:57:50,265 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:57:50,265 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:57:50,265 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:57:50,265 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:57:50,265 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/C/059653d6da58499db49c19ab54892bad as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/059653d6da58499db49c19ab54892bad 2023-03-21 11:57:50,265 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:57:50,265 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:57:50,266 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:50,266 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:50,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:50,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:50,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:50,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:50,266 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:50,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:50,266 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:50,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:50,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:50,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:50,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:50,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:50,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:50,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:50,267 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:50,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:50,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:50,268 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/059653d6da58499db49c19ab54892bad, entries=150, sequenceid=192, filesize=11.8 K 2023-03-21 11:57:50,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:50,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:50,268 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 8dbd4df2f9769e1bcf08d64a8aceb9ad in 1259ms, sequenceid=192, compaction requested=true 2023-03-21 11:57:50,269 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8dbd4df2f9769e1bcf08d64a8aceb9ad: 2023-03-21 11:57:50,269 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-21 11:57:50,269 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-21 11:57:50,269 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-21 11:57:50,269 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-21 11:57:50,269 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-21 11:57:50,269 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:50,269 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:50,269 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:50,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:50,269 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:50,269 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:50,269 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:50,269 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:50,269 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:50,269 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:50,269 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:57:50,269 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:57:50,269 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:50,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:50,270 DEBUG [Thread-2783] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:57:50,270 DEBUG [Thread-2783] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:57:50,270 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:57:50,270 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 38806 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-21 11:57:50,270 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36416 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-21 11:57:50,270 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1912): 8dbd4df2f9769e1bcf08d64a8aceb9ad/A is initiating minor compaction (all files) 2023-03-21 11:57:50,270 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1912): 8dbd4df2f9769e1bcf08d64a8aceb9ad/B is initiating minor compaction (all files) 2023-03-21 11:57:50,270 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8dbd4df2f9769e1bcf08d64a8aceb9ad/A in TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:50,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:50,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:50,270 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/906ec9a526024ace9409a51f7d3dbc5c, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/dceaf26e29fb4b62961d135a5f226aed, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/2abf421417db453f89a497047c79f2f2] into tmpdir=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp, totalSize=37.9 K 2023-03-21 11:57:50,270 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8dbd4df2f9769e1bcf08d64a8aceb9ad/B in TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:50,270 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/349ea59266304913a63e4d1e0a769cd6, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/3a398d0612d04e3bb07f16f19e8a52cf, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/c307cf7252954120a77104dd171f9519] into tmpdir=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp, totalSize=35.6 K 2023-03-21 11:57:50,270 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 906ec9a526024ace9409a51f7d3dbc5c, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=156, earliestPutTs=1679399866433 2023-03-21 11:57:50,271 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 349ea59266304913a63e4d1e0a769cd6, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=156, earliestPutTs=1679399866433 2023-03-21 11:57:50,271 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting dceaf26e29fb4b62961d135a5f226aed, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=175, earliestPutTs=1679399867651 2023-03-21 11:57:50,271 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 3a398d0612d04e3bb07f16f19e8a52cf, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=175, earliestPutTs=1679399867651 2023-03-21 11:57:50,271 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 2abf421417db453f89a497047c79f2f2, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=192, earliestPutTs=1679399868365 2023-03-21 11:57:50,271 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting c307cf7252954120a77104dd171f9519, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=192, earliestPutTs=1679399868365 2023-03-21 11:57:50,275 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8dbd4df2f9769e1bcf08d64a8aceb9ad#A#compaction#381 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 57.69 MB/second 2023-03-21 11:57:50,276 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] throttle.PressureAwareThroughputController(145): 8dbd4df2f9769e1bcf08d64a8aceb9ad#B#compaction#382 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 57.69 MB/second 2023-03-21 11:57:50,370 DEBUG [Thread-2783] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:50,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:50,371 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:57:50,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:57:50,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:57:50,372 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:57:50,372 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:50,372 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:57:50,372 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:57:50,372 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:50,373 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:50,373 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:50,373 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:50,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:50,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:50,373 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:50,373 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:57:50,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:50,374 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:50,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:57:50,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:50,374 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:57:50,374 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:57:50,374 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:57:50,374 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:50,374 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:57:50,374 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:57:50,374 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:57:50,374 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:50,375 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. started... 2023-03-21 11:57:50,375 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8dbd4df2f9769e1bcf08d64a8aceb9ad 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-03-21 11:57:50,375 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dbd4df2f9769e1bcf08d64a8aceb9ad, store=A 2023-03-21 11:57:50,375 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:57:50,375 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dbd4df2f9769e1bcf08d64a8aceb9ad, store=B 2023-03-21 11:57:50,375 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:57:50,375 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dbd4df2f9769e1bcf08d64a8aceb9ad, store=C 2023-03-21 11:57:50,375 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:57:50,684 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/A/1181414af9ad49f19714eb5b930ab75b as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/1181414af9ad49f19714eb5b930ab75b 2023-03-21 11:57:50,689 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8dbd4df2f9769e1bcf08d64a8aceb9ad/A of 8dbd4df2f9769e1bcf08d64a8aceb9ad into 1181414af9ad49f19714eb5b930ab75b(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-21 11:57:50,689 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8dbd4df2f9769e1bcf08d64a8aceb9ad: 2023-03-21 11:57:50,689 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad., storeName=8dbd4df2f9769e1bcf08d64a8aceb9ad/A, priority=13, startTime=1679399870269; duration=0sec 2023-03-21 11:57:50,689 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-21 11:57:50,689 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-21 11:57:50,690 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36416 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-21 11:57:50,690 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1912): 8dbd4df2f9769e1bcf08d64a8aceb9ad/C is initiating minor compaction (all files) 2023-03-21 11:57:50,691 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8dbd4df2f9769e1bcf08d64a8aceb9ad/C in TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:50,691 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/b51026cea1e14b69a357d1f2a48cfbd0, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/b35cf28e8e7146b6899cfa8fc5fe48ec, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/059653d6da58499db49c19ab54892bad] into tmpdir=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp, totalSize=35.6 K 2023-03-21 11:57:50,691 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting b51026cea1e14b69a357d1f2a48cfbd0, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=156, earliestPutTs=1679399866433 2023-03-21 11:57:50,691 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting b35cf28e8e7146b6899cfa8fc5fe48ec, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=175, earliestPutTs=1679399867651 2023-03-21 11:57:50,691 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 059653d6da58499db49c19ab54892bad, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=192, earliestPutTs=1679399868365 2023-03-21 11:57:50,697 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/B/f098f9d7c2a64c2c89b776f2be1b4c5d as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/f098f9d7c2a64c2c89b776f2be1b4c5d 2023-03-21 11:57:50,699 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8dbd4df2f9769e1bcf08d64a8aceb9ad#C#compaction#384 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 57.69 MB/second 2023-03-21 11:57:50,701 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8dbd4df2f9769e1bcf08d64a8aceb9ad/B of 8dbd4df2f9769e1bcf08d64a8aceb9ad into f098f9d7c2a64c2c89b776f2be1b4c5d(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-21 11:57:50,701 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8dbd4df2f9769e1bcf08d64a8aceb9ad: 2023-03-21 11:57:50,701 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad., storeName=8dbd4df2f9769e1bcf08d64a8aceb9ad/B, priority=13, startTime=1679399870269; duration=0sec 2023-03-21 11:57:50,701 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-21 11:57:50,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(9158): Flush requested on 8dbd4df2f9769e1bcf08d64a8aceb9ad 2023-03-21 11:57:50,756 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. as already flushing 2023-03-21 11:57:50,782 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=210 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/A/36329bbf325443dfa9e3a8b822518aa3 2023-03-21 11:57:50,799 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:50,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48284 deadline: 1679399930791, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:50,799 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:50,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48302 deadline: 1679399930794, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:50,801 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=210 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/B/6ad2d455bdc245f9ab2049fc0d4d50ee 2023-03-21 11:57:50,808 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:50,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48336 deadline: 1679399930804, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:50,813 DEBUG [Thread-2779] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8207 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad., hostname=jenkins-hbase20.apache.org,34369,1679399732030, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-21 11:57:50,815 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:50,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48324 deadline: 1679399930810, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:50,817 DEBUG [Thread-2781] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8215 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad., hostname=jenkins-hbase20.apache.org,34369,1679399732030, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-21 11:57:50,903 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:50,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48284 deadline: 1679399930900, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:50,903 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:50,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48302 deadline: 1679399930900, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:51,107 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/C/6401b28e4e6a459baeed257a89acb86b as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/6401b28e4e6a459baeed257a89acb86b 2023-03-21 11:57:51,107 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:51,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48284 deadline: 1679399931104, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:51,107 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:51,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48302 deadline: 1679399931105, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:51,110 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8dbd4df2f9769e1bcf08d64a8aceb9ad/C of 8dbd4df2f9769e1bcf08d64a8aceb9ad into 6401b28e4e6a459baeed257a89acb86b(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-21 11:57:51,110 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8dbd4df2f9769e1bcf08d64a8aceb9ad: 2023-03-21 11:57:51,110 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad., storeName=8dbd4df2f9769e1bcf08d64a8aceb9ad/C, priority=13, startTime=1679399870269; duration=0sec 2023-03-21 11:57:51,110 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-21 11:57:51,212 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:51,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48362 deadline: 1679399931207, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:51,219 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=210 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/C/0d4f9c1fb71f44ab8d48af6ea030f1da 2023-03-21 11:57:51,222 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/A/36329bbf325443dfa9e3a8b822518aa3 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/36329bbf325443dfa9e3a8b822518aa3 2023-03-21 11:57:51,224 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/36329bbf325443dfa9e3a8b822518aa3, entries=150, sequenceid=210, filesize=11.8 K 2023-03-21 11:57:51,225 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/B/6ad2d455bdc245f9ab2049fc0d4d50ee as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/6ad2d455bdc245f9ab2049fc0d4d50ee 2023-03-21 11:57:51,231 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/6ad2d455bdc245f9ab2049fc0d4d50ee, entries=150, sequenceid=210, filesize=11.8 K 2023-03-21 11:57:51,234 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/C/0d4f9c1fb71f44ab8d48af6ea030f1da as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/0d4f9c1fb71f44ab8d48af6ea030f1da 2023-03-21 11:57:51,237 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/0d4f9c1fb71f44ab8d48af6ea030f1da, entries=150, sequenceid=210, filesize=11.8 K 2023-03-21 11:57:51,238 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 8dbd4df2f9769e1bcf08d64a8aceb9ad in 863ms, sequenceid=210, compaction requested=false 2023-03-21 11:57:51,238 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8dbd4df2f9769e1bcf08d64a8aceb9ad: 2023-03-21 11:57:51,238 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:51,238 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:57:51,240 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:57:51,240 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:51,240 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:57:51,240 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:57:51,241 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:51,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:51,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:51,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:51,241 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:51,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:51,242 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:51,242 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:57:51,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:51,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:51,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:51,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:51,242 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:57:51,243 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6d62c9ea[Count = 0] remaining members to acquire global barrier 2023-03-21 11:57:51,243 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:57:51,243 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:51,243 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:51,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:51,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:51,243 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:57:51,243 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:57:51,243 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:57:51,243 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:51,243 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:57:51,245 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:51,245 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:57:51,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:51,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:51,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:51,245 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:51,245 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:57:51,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:51,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:51,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:51,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:51,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:51,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:51,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:51,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:57:51,247 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:57:51,247 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:57:51,247 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:57:51,247 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:57:51,247 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:57:51,247 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:57:51,248 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:51,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:51,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:51,248 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:51,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:51,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:51,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:51,248 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:51,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:51,249 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:51,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:51,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:51,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:51,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:51,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:51,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:51,250 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:51,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:51,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:51,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:51,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:51,251 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:51,251 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:51,251 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:51,251 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:57:51,251 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:57:51,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:51,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:51,251 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:51,251 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:51,252 DEBUG [Thread-2783] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:57:51,251 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:57:51,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:51,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:51,251 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:51,252 DEBUG [Thread-2783] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:57:51,252 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:51,252 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:51,252 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:51,252 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:51,352 DEBUG [Thread-2783] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:51,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:51,353 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:57:51,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:57:51,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:57:51,354 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:57:51,354 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:51,354 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:57:51,354 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:57:51,355 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:51,355 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:51,355 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:51,355 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:51,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:51,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:51,355 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:51,355 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:57:51,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:51,356 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:51,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:57:51,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:51,356 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:57:51,356 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:57:51,356 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:57:51,356 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:57:51,356 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:51,356 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:57:51,357 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:57:51,357 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:57:51,357 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:51,357 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. started... 2023-03-21 11:57:51,357 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8dbd4df2f9769e1bcf08d64a8aceb9ad 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-03-21 11:57:51,358 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dbd4df2f9769e1bcf08d64a8aceb9ad, store=A 2023-03-21 11:57:51,358 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:57:51,358 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dbd4df2f9769e1bcf08d64a8aceb9ad, store=B 2023-03-21 11:57:51,358 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:57:51,358 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dbd4df2f9769e1bcf08d64a8aceb9ad, store=C 2023-03-21 11:57:51,358 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:57:51,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(9158): Flush requested on 8dbd4df2f9769e1bcf08d64a8aceb9ad 2023-03-21 11:57:51,413 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. as already flushing 2023-03-21 11:57:51,449 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:51,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48302 deadline: 1679399931446, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:51,454 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:51,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48284 deadline: 1679399931449, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:51,551 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:51,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48302 deadline: 1679399931550, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:51,556 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:51,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48284 deadline: 1679399931554, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:51,755 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:51,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48302 deadline: 1679399931752, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:51,760 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:51,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48284 deadline: 1679399931757, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:51,770 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=227 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/A/2a34d28534194ec2ac7a4a62a0030e51 2023-03-21 11:57:52,061 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:52,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48302 deadline: 1679399932058, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:52,065 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:52,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48284 deadline: 1679399932062, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:52,179 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=227 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/B/d0f86ae6ccc94bf39fdd3c3b2a8d29c6 2023-03-21 11:57:52,568 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:52,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48302 deadline: 1679399932565, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:52,571 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:52,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48284 deadline: 1679399932567, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:52,587 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=227 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/C/966966a35deb43208cb7cb1f704fdd06 2023-03-21 11:57:52,590 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/A/2a34d28534194ec2ac7a4a62a0030e51 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/2a34d28534194ec2ac7a4a62a0030e51 2023-03-21 11:57:52,593 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/2a34d28534194ec2ac7a4a62a0030e51, entries=150, sequenceid=227, filesize=11.8 K 2023-03-21 11:57:52,594 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/B/d0f86ae6ccc94bf39fdd3c3b2a8d29c6 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/d0f86ae6ccc94bf39fdd3c3b2a8d29c6 2023-03-21 11:57:52,597 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/d0f86ae6ccc94bf39fdd3c3b2a8d29c6, entries=150, sequenceid=227, filesize=11.8 K 2023-03-21 11:57:52,598 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/C/966966a35deb43208cb7cb1f704fdd06 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/966966a35deb43208cb7cb1f704fdd06 2023-03-21 11:57:52,601 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/966966a35deb43208cb7cb1f704fdd06, entries=150, sequenceid=227, filesize=11.8 K 2023-03-21 11:57:52,602 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 8dbd4df2f9769e1bcf08d64a8aceb9ad in 1245ms, sequenceid=227, compaction requested=true 2023-03-21 11:57:52,602 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8dbd4df2f9769e1bcf08d64a8aceb9ad: 2023-03-21 11:57:52,602 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:52,602 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:57:52,602 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:57:52,602 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:52,603 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:57:52,603 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:57:52,604 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:52,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:52,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:52,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:52,604 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:52,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:52,604 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:52,604 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:57:52,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:52,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:52,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:52,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:52,605 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:57:52,605 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2fdc3542[Count = 0] remaining members to acquire global barrier 2023-03-21 11:57:52,605 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:57:52,605 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:52,606 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:52,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:52,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:52,606 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:57:52,606 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:57:52,606 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:57:52,606 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:52,606 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:57:52,607 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:52,607 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:57:52,607 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:52,607 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:57:52,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:52,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:52,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:52,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:52,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:52,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:52,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:52,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:52,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:52,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:52,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:57:52,609 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:57:52,609 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:57:52,609 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:57:52,609 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:57:52,609 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:57:52,609 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:57:52,612 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:52,612 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:52,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:52,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:52,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:52,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:52,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:52,612 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:52,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:52,612 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:52,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:52,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:52,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:52,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:52,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:52,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:52,614 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:52,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:52,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:52,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:52,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:52,616 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:52,616 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:52,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:52,616 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:52,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:52,616 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:52,616 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:57:52,616 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:52,616 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:57:52,616 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:57:52,616 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:52,616 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:52,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:52,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:52,616 DEBUG [Thread-2783] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:57:52,616 DEBUG [Thread-2783] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:57:52,616 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:52,617 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:52,617 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:52,717 DEBUG [Thread-2783] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:52,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:52,718 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:57:52,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:57:52,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:57:52,719 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:57:52,719 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:52,719 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:57:52,719 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:57:52,720 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:52,720 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:52,720 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:52,720 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:52,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:52,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:52,721 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:52,721 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:57:52,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:52,721 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:52,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:57:52,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:52,721 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:57:52,722 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:57:52,722 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:57:52,722 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:57:52,722 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:52,722 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:57:52,722 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:57:52,722 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:57:52,722 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:52,722 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. started... 2023-03-21 11:57:52,722 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8dbd4df2f9769e1bcf08d64a8aceb9ad 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-03-21 11:57:52,722 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dbd4df2f9769e1bcf08d64a8aceb9ad, store=A 2023-03-21 11:57:52,722 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:57:52,723 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dbd4df2f9769e1bcf08d64a8aceb9ad, store=B 2023-03-21 11:57:52,723 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:57:52,723 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dbd4df2f9769e1bcf08d64a8aceb9ad, store=C 2023-03-21 11:57:52,723 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:57:53,132 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=245 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/A/43d97de7b1554db4861deaa29dc22a0d 2023-03-21 11:57:53,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(9158): Flush requested on 8dbd4df2f9769e1bcf08d64a8aceb9ad 2023-03-21 11:57:53,220 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. as already flushing 2023-03-21 11:57:53,285 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:53,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48362 deadline: 1679399933278, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:53,388 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:53,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48362 deadline: 1679399933386, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:53,539 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=245 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/B/867fe675231f425a94dfce964a19a4b2 2023-03-21 11:57:53,547 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=245 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/C/7245a8d75dab4ef59f461ab0deab300f 2023-03-21 11:57:53,550 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/A/43d97de7b1554db4861deaa29dc22a0d as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/43d97de7b1554db4861deaa29dc22a0d 2023-03-21 11:57:53,552 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/43d97de7b1554db4861deaa29dc22a0d, entries=150, sequenceid=245, filesize=11.8 K 2023-03-21 11:57:53,553 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/B/867fe675231f425a94dfce964a19a4b2 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/867fe675231f425a94dfce964a19a4b2 2023-03-21 11:57:53,555 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/867fe675231f425a94dfce964a19a4b2, entries=150, sequenceid=245, filesize=11.8 K 2023-03-21 11:57:53,557 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/C/7245a8d75dab4ef59f461ab0deab300f as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/7245a8d75dab4ef59f461ab0deab300f 2023-03-21 11:57:53,561 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/7245a8d75dab4ef59f461ab0deab300f, entries=150, sequenceid=245, filesize=11.8 K 2023-03-21 11:57:53,562 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 8dbd4df2f9769e1bcf08d64a8aceb9ad in 840ms, sequenceid=245, compaction requested=true 2023-03-21 11:57:53,562 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8dbd4df2f9769e1bcf08d64a8aceb9ad: 2023-03-21 11:57:53,562 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:53,562 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:57:53,562 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:57:53,562 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:53,562 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:57:53,562 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:57:53,563 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:53,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:53,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:53,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:53,563 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:53,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:53,564 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:53,564 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:57:53,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:53,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:53,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:53,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:53,565 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:57:53,565 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@33700bd2[Count = 0] remaining members to acquire global barrier 2023-03-21 11:57:53,565 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:57:53,565 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:53,565 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:53,565 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:57:53,566 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:53,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:53,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:53,566 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:57:53,566 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:57:53,566 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:57:53,567 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:57:53,567 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:53,567 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:53,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:53,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:53,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:53,567 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:57:53,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:53,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:53,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:53,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:53,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:53,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:53,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:53,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:57:53,569 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:57:53,569 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:57:53,569 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:57:53,569 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:57:53,569 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:57:53,569 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:57:53,569 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:53,569 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:53,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:53,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:53,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:53,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:53,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:53,570 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:53,570 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:53,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:53,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:53,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:53,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:53,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:53,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:53,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:53,571 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:53,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:53,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:53,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:53,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:53,572 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:53,572 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:53,572 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:53,572 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:57:53,572 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:57:53,572 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:57:53,572 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:53,572 DEBUG [Thread-2783] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:57:53,572 DEBUG [Thread-2783] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:57:53,572 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:53,572 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:53,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:53,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:53,572 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:53,572 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:53,573 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:53,573 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:53,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:53,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:53,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(9158): Flush requested on 8dbd4df2f9769e1bcf08d64a8aceb9ad 2023-03-21 11:57:53,577 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8dbd4df2f9769e1bcf08d64a8aceb9ad 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-03-21 11:57:53,577 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dbd4df2f9769e1bcf08d64a8aceb9ad, store=A 2023-03-21 11:57:53,577 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:57:53,577 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dbd4df2f9769e1bcf08d64a8aceb9ad, store=B 2023-03-21 11:57:53,578 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:57:53,578 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dbd4df2f9769e1bcf08d64a8aceb9ad, store=C 2023-03-21 11:57:53,578 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:57:53,613 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:53,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48362 deadline: 1679399933604, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:53,614 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:53,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48284 deadline: 1679399933610, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:53,621 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:53,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48302 deadline: 1679399933614, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:53,672 DEBUG [Thread-2783] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:53,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:53,674 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:57:53,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:57:53,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:57:53,675 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:57:53,675 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:53,675 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:57:53,675 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:57:53,676 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:53,676 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:53,676 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:53,676 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:53,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:53,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:53,676 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:53,677 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:57:53,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:53,677 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:53,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:57:53,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:53,677 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:57:53,677 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:57:53,677 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:57:53,677 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:57:53,677 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:53,677 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:57:53,678 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:57:53,678 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:57:53,678 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:53,678 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. started... 2023-03-21 11:57:53,678 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. as already flushing 2023-03-21 11:57:53,678 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:53,678 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:57:53,678 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:57:53,678 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:53,678 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:57:53,678 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:57:53,679 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:53,679 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:53,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:53,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:53,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:53,679 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:53,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:53,679 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:57:53,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:53,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:53,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:53,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:53,680 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:57:53,680 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a7144fa[Count = 0] remaining members to acquire global barrier 2023-03-21 11:57:53,680 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:57:53,680 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:53,681 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:53,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:53,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:53,681 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:57:53,681 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:53,681 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:57:53,681 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:57:53,681 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:57:53,681 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:53,681 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:57:53,681 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:53,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:53,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:53,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:53,682 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:57:53,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:53,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:53,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:53,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:53,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:53,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:53,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:53,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:57:53,683 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:57:53,683 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:57:53,683 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:57:53,683 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:57:53,683 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:57:53,683 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:57:53,684 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:53,684 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:53,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:53,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:53,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:53,684 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:53,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:53,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:53,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:53,684 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:53,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:53,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:53,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:53,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:53,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:53,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:53,685 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:53,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:53,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:53,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:53,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:53,688 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:53,688 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:53,688 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:53,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:53,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:53,688 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:57:53,688 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:57:53,688 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:53,688 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:53,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:53,688 DEBUG [Thread-2783] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:57:53,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:53,688 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:57:53,688 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:53,688 DEBUG [Thread-2783] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:57:53,688 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:53,688 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:53,688 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:53,688 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:53,719 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:53,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48362 deadline: 1679399933714, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:53,719 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:53,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48284 deadline: 1679399933715, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:53,726 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:53,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48302 deadline: 1679399933722, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:53,788 DEBUG [Thread-2783] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:53,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:53,790 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:57:53,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:57:53,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:57:53,791 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:57:53,791 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:53,791 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:57:53,791 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:57:53,791 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:53,791 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:53,791 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:53,792 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:53,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:53,792 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:53,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:53,792 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:53,792 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:57:53,792 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:53,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:57:53,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:53,792 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:57:53,792 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:57:53,793 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:57:53,793 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:57:53,793 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:53,793 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:57:53,793 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:57:53,793 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:57:53,793 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:53,793 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. started... 2023-03-21 11:57:53,793 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. as already flushing 2023-03-21 11:57:53,793 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:53,793 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:57:53,793 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:57:53,793 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:53,793 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:57:53,793 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:57:53,794 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:53,794 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:53,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:53,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:53,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:53,794 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:53,794 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:57:53,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:53,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:53,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:53,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:53,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:53,795 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:57:53,795 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1127fdc7[Count = 0] remaining members to acquire global barrier 2023-03-21 11:57:53,795 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:57:53,795 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:53,796 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:53,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:53,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:53,796 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:57:53,796 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:57:53,796 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:57:53,796 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:53,796 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:57:53,797 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:53,797 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:57:53,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:53,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:53,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:53,797 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:53,797 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:57:53,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:53,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:53,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:53,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:53,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:53,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:53,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:53,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:57:53,798 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:57:53,798 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:57:53,798 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:57:53,799 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:57:53,799 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:57:53,799 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:57:53,801 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:53,801 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:53,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:53,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:53,801 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:53,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:53,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:53,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:53,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:53,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:53,801 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:53,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:53,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:53,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:53,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:53,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:53,802 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:53,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:53,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:53,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:53,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:53,803 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:53,804 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:53,804 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:53,804 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:53,804 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:53,804 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:53,804 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:53,804 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:57:53,804 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:57:53,804 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:53,804 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:57:53,804 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:53,804 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:53,804 DEBUG [Thread-2783] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:57:53,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:53,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:53,804 DEBUG [Thread-2783] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:57:53,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:53,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:53,904 DEBUG [Thread-2783] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:53,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:53,906 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:57:53,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:57:53,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:57:53,907 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:57:53,907 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:53,907 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:57:53,907 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:57:53,907 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:53,907 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:53,908 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:53,908 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:53,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:53,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:53,908 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:53,908 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:57:53,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:53,909 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:53,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:57:53,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:53,909 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:57:53,909 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:57:53,909 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:57:53,909 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:57:53,909 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:53,909 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:57:53,909 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:57:53,909 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:57:53,909 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:53,909 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. started... 2023-03-21 11:57:53,909 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. as already flushing 2023-03-21 11:57:53,910 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:53,910 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:57:53,910 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:57:53,910 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:53,910 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:57:53,910 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:57:53,910 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:53,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:53,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:53,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:53,910 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:53,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:53,911 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:53,911 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:57:53,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:53,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:53,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:53,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:53,912 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:57:53,912 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f1fac84[Count = 0] remaining members to acquire global barrier 2023-03-21 11:57:53,912 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:57:53,912 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:53,912 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:53,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:53,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:53,912 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:57:53,912 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:57:53,912 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:57:53,912 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:53,913 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:57:53,913 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:53,913 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:57:53,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:53,913 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:53,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:53,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:53,913 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:57:53,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:53,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:53,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:53,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:53,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:53,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:53,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:53,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:57:53,915 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:57:53,915 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:57:53,915 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:57:53,915 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:57:53,915 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:57:53,915 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:57:53,916 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:53,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:53,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:53,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:53,916 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:53,916 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:53,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:53,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:53,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:53,916 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:53,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:53,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:53,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:53,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:53,917 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:53,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:53,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:53,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:53,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:53,919 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:53,919 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:53,919 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:53,919 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:57:53,919 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:57:53,919 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:53,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:53,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:53,919 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:53,919 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:53,919 DEBUG [Thread-2783] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:57:53,919 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:53,919 DEBUG [Thread-2783] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:57:53,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:53,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:53,919 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:57:53,919 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:53,920 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:53,920 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:53,924 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:53,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48362 deadline: 1679399933921, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:53,924 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:53,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48284 deadline: 1679399933921, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:53,932 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:53,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48302 deadline: 1679399933928, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:53,983 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=261 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/A/3968a80e65f849fc8ad24bf53fb688a4 2023-03-21 11:57:53,991 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=261 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/B/400c2550f06f49cb90379e0342216521 2023-03-21 11:57:53,997 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=261 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/C/cc95426bd4c94e92bf5a42ddd60409b8 2023-03-21 11:57:54,000 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/A/3968a80e65f849fc8ad24bf53fb688a4 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/3968a80e65f849fc8ad24bf53fb688a4 2023-03-21 11:57:54,003 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/3968a80e65f849fc8ad24bf53fb688a4, entries=200, sequenceid=261, filesize=14.2 K 2023-03-21 11:57:54,003 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/B/400c2550f06f49cb90379e0342216521 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/400c2550f06f49cb90379e0342216521 2023-03-21 11:57:54,006 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/400c2550f06f49cb90379e0342216521, entries=150, sequenceid=261, filesize=11.9 K 2023-03-21 11:57:54,007 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/C/cc95426bd4c94e92bf5a42ddd60409b8 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/cc95426bd4c94e92bf5a42ddd60409b8 2023-03-21 11:57:54,011 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/cc95426bd4c94e92bf5a42ddd60409b8, entries=150, sequenceid=261, filesize=11.9 K 2023-03-21 11:57:54,011 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 8dbd4df2f9769e1bcf08d64a8aceb9ad in 435ms, sequenceid=261, compaction requested=true 2023-03-21 11:57:54,011 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8dbd4df2f9769e1bcf08d64a8aceb9ad: 2023-03-21 11:57:54,012 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-21 11:57:54,012 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-21 11:57:54,012 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-21 11:57:54,012 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-21 11:57:54,012 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-21 11:57:54,013 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 63130 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-21 11:57:54,013 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1912): 8dbd4df2f9769e1bcf08d64a8aceb9ad/A is initiating minor compaction (all files) 2023-03-21 11:57:54,013 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8dbd4df2f9769e1bcf08d64a8aceb9ad/A in TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:54,013 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/1181414af9ad49f19714eb5b930ab75b, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/36329bbf325443dfa9e3a8b822518aa3, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/2a34d28534194ec2ac7a4a62a0030e51, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/43d97de7b1554db4861deaa29dc22a0d, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/3968a80e65f849fc8ad24bf53fb688a4] into tmpdir=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp, totalSize=61.7 K 2023-03-21 11:57:54,013 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 60740 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-21 11:57:54,013 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1912): 8dbd4df2f9769e1bcf08d64a8aceb9ad/B is initiating minor compaction (all files) 2023-03-21 11:57:54,013 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 1181414af9ad49f19714eb5b930ab75b, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=192, earliestPutTs=1679399868365 2023-03-21 11:57:54,013 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8dbd4df2f9769e1bcf08d64a8aceb9ad/B in TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:54,013 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/f098f9d7c2a64c2c89b776f2be1b4c5d, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/6ad2d455bdc245f9ab2049fc0d4d50ee, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/d0f86ae6ccc94bf39fdd3c3b2a8d29c6, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/867fe675231f425a94dfce964a19a4b2, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/400c2550f06f49cb90379e0342216521] into tmpdir=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp, totalSize=59.3 K 2023-03-21 11:57:54,013 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 36329bbf325443dfa9e3a8b822518aa3, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=210, earliestPutTs=1679399869054 2023-03-21 11:57:54,014 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting f098f9d7c2a64c2c89b776f2be1b4c5d, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=192, earliestPutTs=1679399868365 2023-03-21 11:57:54,014 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 2a34d28534194ec2ac7a4a62a0030e51, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=227, earliestPutTs=1679399870773 2023-03-21 11:57:54,014 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 6ad2d455bdc245f9ab2049fc0d4d50ee, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=210, earliestPutTs=1679399869054 2023-03-21 11:57:54,014 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 43d97de7b1554db4861deaa29dc22a0d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=245, earliestPutTs=1679399871434 2023-03-21 11:57:54,014 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting d0f86ae6ccc94bf39fdd3c3b2a8d29c6, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=227, earliestPutTs=1679399870773 2023-03-21 11:57:54,014 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 3968a80e65f849fc8ad24bf53fb688a4, keycount=200, bloomtype=ROW, size=14.2 K, encoding=NONE, compression=NONE, seqNum=261, earliestPutTs=1679399873261 2023-03-21 11:57:54,014 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 867fe675231f425a94dfce964a19a4b2, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=245, earliestPutTs=1679399871434 2023-03-21 11:57:54,015 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 400c2550f06f49cb90379e0342216521, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=261, earliestPutTs=1679399873277 2023-03-21 11:57:54,020 DEBUG [Thread-2783] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:54,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:54,021 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:57:54,021 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8dbd4df2f9769e1bcf08d64a8aceb9ad#A#compaction#396 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 57.69 MB/second 2023-03-21 11:57:54,021 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] throttle.PressureAwareThroughputController(145): 8dbd4df2f9769e1bcf08d64a8aceb9ad#B#compaction#397 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 57.69 MB/second 2023-03-21 11:57:54,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:57:54,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:57:54,022 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:57:54,022 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:54,022 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:57:54,022 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:57:54,023 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:54,023 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:54,023 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:54,024 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:54,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:54,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:54,024 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:54,024 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:57:54,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:54,024 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:54,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:57:54,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:54,024 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:57:54,026 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:57:54,026 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:57:54,026 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:57:54,026 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:54,026 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:57:54,026 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:57:54,026 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:57:54,026 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:54,026 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. started... 2023-03-21 11:57:54,026 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8dbd4df2f9769e1bcf08d64a8aceb9ad 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-03-21 11:57:54,027 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dbd4df2f9769e1bcf08d64a8aceb9ad, store=A 2023-03-21 11:57:54,027 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:57:54,027 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dbd4df2f9769e1bcf08d64a8aceb9ad, store=B 2023-03-21 11:57:54,027 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:57:54,027 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dbd4df2f9769e1bcf08d64a8aceb9ad, store=C 2023-03-21 11:57:54,027 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:57:54,033 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/B/8d345ea0ff1c4b819d11944cd5014728 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/8d345ea0ff1c4b819d11944cd5014728 2023-03-21 11:57:54,036 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/A/a0ce75904f18473988823ac43e192f8f as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/a0ce75904f18473988823ac43e192f8f 2023-03-21 11:57:54,037 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=277 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/A/71ea227e90e74850a2392ce48309483a 2023-03-21 11:57:54,037 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 8dbd4df2f9769e1bcf08d64a8aceb9ad/B of 8dbd4df2f9769e1bcf08d64a8aceb9ad into 8d345ea0ff1c4b819d11944cd5014728(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-21 11:57:54,037 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8dbd4df2f9769e1bcf08d64a8aceb9ad: 2023-03-21 11:57:54,037 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad., storeName=8dbd4df2f9769e1bcf08d64a8aceb9ad/B, priority=11, startTime=1679399874012; duration=0sec 2023-03-21 11:57:54,037 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-21 11:57:54,037 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-21 11:57:54,039 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 60740 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-21 11:57:54,039 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1912): 8dbd4df2f9769e1bcf08d64a8aceb9ad/C is initiating minor compaction (all files) 2023-03-21 11:57:54,039 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8dbd4df2f9769e1bcf08d64a8aceb9ad/C in TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:54,039 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/6401b28e4e6a459baeed257a89acb86b, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/0d4f9c1fb71f44ab8d48af6ea030f1da, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/966966a35deb43208cb7cb1f704fdd06, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/7245a8d75dab4ef59f461ab0deab300f, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/cc95426bd4c94e92bf5a42ddd60409b8] into tmpdir=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp, totalSize=59.3 K 2023-03-21 11:57:54,039 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 6401b28e4e6a459baeed257a89acb86b, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=192, earliestPutTs=1679399868365 2023-03-21 11:57:54,040 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 0d4f9c1fb71f44ab8d48af6ea030f1da, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=210, earliestPutTs=1679399869054 2023-03-21 11:57:54,040 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 966966a35deb43208cb7cb1f704fdd06, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=227, earliestPutTs=1679399870773 2023-03-21 11:57:54,040 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 7245a8d75dab4ef59f461ab0deab300f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=245, earliestPutTs=1679399871434 2023-03-21 11:57:54,041 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting cc95426bd4c94e92bf5a42ddd60409b8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=261, earliestPutTs=1679399873277 2023-03-21 11:57:54,041 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 8dbd4df2f9769e1bcf08d64a8aceb9ad/A of 8dbd4df2f9769e1bcf08d64a8aceb9ad into a0ce75904f18473988823ac43e192f8f(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-21 11:57:54,041 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8dbd4df2f9769e1bcf08d64a8aceb9ad: 2023-03-21 11:57:54,041 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad., storeName=8dbd4df2f9769e1bcf08d64a8aceb9ad/A, priority=11, startTime=1679399874012; duration=0sec 2023-03-21 11:57:54,041 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-21 11:57:54,055 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] throttle.PressureAwareThroughputController(145): 8dbd4df2f9769e1bcf08d64a8aceb9ad#C#compaction#400 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 57.69 MB/second 2023-03-21 11:57:54,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(9158): Flush requested on 8dbd4df2f9769e1bcf08d64a8aceb9ad 2023-03-21 11:57:54,228 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. as already flushing 2023-03-21 11:57:54,258 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:54,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48362 deadline: 1679399934253, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:54,258 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:54,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48302 deadline: 1679399934253, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:54,263 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:54,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48284 deadline: 1679399934254, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:54,362 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:54,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48362 deadline: 1679399934359, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:54,363 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:54,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48302 deadline: 1679399934359, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:54,368 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:54,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48284 deadline: 1679399934364, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:54,460 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=277 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/B/1a3eddc851604db9b8524ea5b849a2f6 2023-03-21 11:57:54,467 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=277 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/C/54769945dc7f459d9c2ad0de3690f9e4 2023-03-21 11:57:54,471 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/C/8e3e6a1fac1d44eaa56ce057b6fbffc9 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/8e3e6a1fac1d44eaa56ce057b6fbffc9 2023-03-21 11:57:54,471 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/A/71ea227e90e74850a2392ce48309483a as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/71ea227e90e74850a2392ce48309483a 2023-03-21 11:57:54,474 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/71ea227e90e74850a2392ce48309483a, entries=150, sequenceid=277, filesize=11.9 K 2023-03-21 11:57:54,475 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/B/1a3eddc851604db9b8524ea5b849a2f6 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/1a3eddc851604db9b8524ea5b849a2f6 2023-03-21 11:57:54,475 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 8dbd4df2f9769e1bcf08d64a8aceb9ad/C of 8dbd4df2f9769e1bcf08d64a8aceb9ad into 8e3e6a1fac1d44eaa56ce057b6fbffc9(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-21 11:57:54,475 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8dbd4df2f9769e1bcf08d64a8aceb9ad: 2023-03-21 11:57:54,475 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad., storeName=8dbd4df2f9769e1bcf08d64a8aceb9ad/C, priority=11, startTime=1679399874012; duration=0sec 2023-03-21 11:57:54,475 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-21 11:57:54,477 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/1a3eddc851604db9b8524ea5b849a2f6, entries=150, sequenceid=277, filesize=11.9 K 2023-03-21 11:57:54,478 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/C/54769945dc7f459d9c2ad0de3690f9e4 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/54769945dc7f459d9c2ad0de3690f9e4 2023-03-21 11:57:54,480 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/54769945dc7f459d9c2ad0de3690f9e4, entries=150, sequenceid=277, filesize=11.9 K 2023-03-21 11:57:54,481 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 8dbd4df2f9769e1bcf08d64a8aceb9ad in 455ms, sequenceid=277, compaction requested=false 2023-03-21 11:57:54,481 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8dbd4df2f9769e1bcf08d64a8aceb9ad: 2023-03-21 11:57:54,481 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:54,481 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:57:54,481 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:57:54,481 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:54,481 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:57:54,481 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:57:54,482 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:54,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:54,482 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:54,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:54,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:54,482 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:54,482 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:57:54,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:54,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:54,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:54,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:54,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:54,485 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:57:54,485 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c02122e[Count = 0] remaining members to acquire global barrier 2023-03-21 11:57:54,485 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:57:54,485 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:54,485 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:54,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:54,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:54,486 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:57:54,486 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:57:54,486 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:54,486 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:57:54,486 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:57:54,486 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:57:54,486 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:54,486 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:54,486 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:57:54,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:54,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:54,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:54,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:54,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:54,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:54,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:54,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:54,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:54,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:54,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:57:54,490 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:57:54,490 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:57:54,490 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:57:54,490 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:57:54,490 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:57:54,490 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:57:54,494 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:54,494 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:54,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:54,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:54,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:54,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:54,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:54,494 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:54,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:54,494 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:54,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:54,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:54,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:54,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:54,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:54,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:54,497 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:54,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:54,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:54,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:54,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:54,498 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:54,499 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:54,499 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:54,499 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:54,499 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:54,499 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:54,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:54,499 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:54,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:54,499 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:57:54,499 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:57:54,499 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:54,499 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:54,499 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:54,499 DEBUG [Thread-2783] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:57:54,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:54,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:54,499 DEBUG [Thread-2783] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:57:54,499 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:57:54,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(9158): Flush requested on 8dbd4df2f9769e1bcf08d64a8aceb9ad 2023-03-21 11:57:54,566 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8dbd4df2f9769e1bcf08d64a8aceb9ad 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-03-21 11:57:54,566 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dbd4df2f9769e1bcf08d64a8aceb9ad, store=A 2023-03-21 11:57:54,566 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:57:54,566 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dbd4df2f9769e1bcf08d64a8aceb9ad, store=B 2023-03-21 11:57:54,566 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:57:54,566 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dbd4df2f9769e1bcf08d64a8aceb9ad, store=C 2023-03-21 11:57:54,566 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:57:54,572 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=297 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/A/ebd74e81e764452f8fa946cf71020c40 2023-03-21 11:57:54,587 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:54,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48362 deadline: 1679399934582, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:54,591 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:54,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48302 deadline: 1679399934586, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:54,591 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:54,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48284 deadline: 1679399934587, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:54,599 DEBUG [Thread-2783] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:54,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:54,601 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:57:54,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:57:54,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:57:54,602 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:57:54,602 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:54,602 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:57:54,602 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:57:54,602 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:54,602 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:54,603 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:54,603 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:54,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:54,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:54,603 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:54,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:54,603 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:57:54,603 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:54,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:57:54,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:54,604 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:57:54,604 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:57:54,604 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:57:54,604 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:57:54,604 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:54,604 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:57:54,604 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:57:54,604 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:57:54,604 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:54,604 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. started... 2023-03-21 11:57:54,604 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. as already flushing 2023-03-21 11:57:54,604 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:54,604 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:57:54,604 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:57:54,604 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:54,604 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:57:54,605 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:57:54,605 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:54,605 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:54,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:54,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:54,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:54,605 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:54,606 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:57:54,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:54,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:54,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:54,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:54,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:54,606 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:57:54,606 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@cb4ddc0[Count = 0] remaining members to acquire global barrier 2023-03-21 11:57:54,606 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:57:54,606 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:54,607 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:54,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:54,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:54,607 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:57:54,607 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:54,607 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:57:54,607 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:57:54,607 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:57:54,608 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:57:54,608 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:54,608 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:54,608 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:57:54,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:54,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:54,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:54,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:54,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:54,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:54,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:54,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:54,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:54,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:54,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:57:54,610 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:57:54,610 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:57:54,610 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:57:54,610 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:57:54,610 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:57:54,610 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:57:54,610 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:54,611 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:54,611 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:54,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:54,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:54,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:54,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:54,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:54,611 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:54,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:54,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:54,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:54,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:54,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:54,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:54,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:54,612 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:54,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:54,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:54,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:54,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:54,613 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:54,613 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:54,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:54,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:54,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:54,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:54,614 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:54,615 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:54,615 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:54,615 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:54,615 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:54,615 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:57:54,615 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:57:54,615 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:54,615 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:54,615 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:57:54,615 DEBUG [Thread-2783] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:57:54,615 DEBUG [Thread-2783] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:57:54,615 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:54,689 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:54,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48362 deadline: 1679399934688, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:54,695 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:54,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48302 deadline: 1679399934692, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:54,695 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:54,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48284 deadline: 1679399934692, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:54,715 DEBUG [Thread-2783] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:54,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:54,717 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:57:54,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:57:54,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:57:54,718 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:57:54,718 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:54,718 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:57:54,718 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:57:54,718 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:54,718 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:54,719 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:54,719 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:54,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:54,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:54,719 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:54,719 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:57:54,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:54,719 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:54,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:57:54,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:54,719 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:57:54,720 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:57:54,720 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:57:54,720 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:57:54,720 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:54,720 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:57:54,720 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:57:54,720 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:57:54,720 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:54,720 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. started... 2023-03-21 11:57:54,720 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. as already flushing 2023-03-21 11:57:54,720 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:54,720 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:57:54,720 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:57:54,720 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:54,720 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:57:54,720 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:57:54,721 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:54,721 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:54,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:54,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:54,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:54,721 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:54,721 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:57:54,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:54,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:54,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:54,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:54,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:54,722 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:57:54,722 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2253c054[Count = 0] remaining members to acquire global barrier 2023-03-21 11:57:54,722 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:57:54,722 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:54,723 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:54,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:54,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:54,723 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:57:54,723 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:57:54,723 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:57:54,723 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:54,723 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:57:54,724 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:54,724 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:57:54,724 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:54,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:54,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:54,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:54,724 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:57:54,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:54,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:54,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:54,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:54,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:54,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:54,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:54,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:57:54,725 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:57:54,725 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:57:54,725 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:57:54,726 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:57:54,726 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:57:54,726 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:57:54,726 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:54,726 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:54,727 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:54,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:54,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:54,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:54,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:54,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:54,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:54,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:54,727 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:54,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:54,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:54,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:54,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:54,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:54,728 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:54,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:54,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:54,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:54,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:54,729 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:54,729 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:54,729 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:54,729 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:54,729 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:57:54,729 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:57:54,729 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:57:54,729 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:54,729 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:54,729 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:54,729 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:54,729 DEBUG [Thread-2783] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:57:54,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:54,730 DEBUG [Thread-2783] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:57:54,730 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:54,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:54,730 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:54,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:54,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:54,830 DEBUG [Thread-2783] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:54,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:54,831 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:57:54,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:57:54,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:57:54,832 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:57:54,832 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:54,832 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:57:54,832 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:57:54,833 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:54,833 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:54,833 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:54,833 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:54,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:54,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:54,834 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:54,834 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:57:54,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:54,834 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:54,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:57:54,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:54,834 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:57:54,834 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:57:54,834 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:57:54,835 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:57:54,835 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:54,835 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:57:54,840 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:57:54,840 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:57:54,840 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:54,840 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. started... 2023-03-21 11:57:54,840 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. as already flushing 2023-03-21 11:57:54,840 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:54,840 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:57:54,840 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:57:54,840 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:54,840 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:57:54,841 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:57:54,848 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:54,848 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:54,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:54,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:54,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:54,848 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:54,848 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:57:54,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:54,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:54,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:54,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:54,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:54,849 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:57:54,850 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@8a2316[Count = 0] remaining members to acquire global barrier 2023-03-21 11:57:54,850 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:57:54,850 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:54,852 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:54,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:54,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:54,853 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:57:54,853 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:57:54,853 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:57:54,853 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:54,853 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:57:54,853 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:54,853 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:57:54,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:54,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:54,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:54,853 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:54,854 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:57:54,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:54,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:54,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:54,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:54,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:54,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:54,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:54,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:57:54,855 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:57:54,855 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:57:54,855 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:57:54,855 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:57:54,855 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:57:54,856 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:57:54,856 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:54,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:54,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:54,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:54,856 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:54,857 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:54,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:54,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:54,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:54,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:54,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:54,857 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:54,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:54,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:54,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:54,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:54,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:54,858 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:54,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:54,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:54,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:54,861 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:54,861 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:54,861 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:54,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:54,861 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:57:54,861 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:54,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:54,861 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:57:54,861 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:57:54,861 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:54,861 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:54,861 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:54,861 DEBUG [Thread-2783] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:57:54,861 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:54,861 DEBUG [Thread-2783] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:57:54,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:54,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:54,861 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:54,861 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:54,895 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:54,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48362 deadline: 1679399934892, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:54,900 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:54,900 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:54,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48302 deadline: 1679399934896, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:54,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48284 deadline: 1679399934896, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:54,961 DEBUG [Thread-2783] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:54,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:54,963 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:57:54,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:57:54,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:57:54,964 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:57:54,964 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:54,964 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:57:54,964 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:57:54,964 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:54,964 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:54,966 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:54,966 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:54,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:54,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:54,966 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:54,966 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:57:54,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:54,966 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:54,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:57:54,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:54,966 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:57:54,967 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:57:54,967 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:57:54,967 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:57:54,967 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:54,967 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:57:54,967 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:57:54,967 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:57:54,967 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:54,967 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. started... 2023-03-21 11:57:54,967 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. as already flushing 2023-03-21 11:57:54,967 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:54,967 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:57:54,967 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:57:54,967 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:54,967 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:57:54,967 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:57:54,968 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:54,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:54,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:54,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:54,968 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:54,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:54,968 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:54,969 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:57:54,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:54,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:54,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:54,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:54,969 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:57:54,969 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c65d24b[Count = 0] remaining members to acquire global barrier 2023-03-21 11:57:54,969 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:57:54,969 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:54,970 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:54,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:54,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:54,970 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:57:54,970 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:57:54,970 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:54,970 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:57:54,970 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:57:54,971 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:57:54,971 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:54,971 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:57:54,971 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:54,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:54,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:54,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:54,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:54,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:54,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:54,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:54,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:54,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:54,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:54,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:57:54,973 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:57:54,973 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:57:54,973 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:57:54,973 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:57:54,973 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:57:54,973 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:57:54,974 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:54,974 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:54,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:54,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:54,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:54,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:54,974 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:54,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:54,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:54,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:54,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:54,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:54,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:54,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:54,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:54,975 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:54,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:54,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:54,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:54,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:54,976 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:54,977 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:54,977 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:54,977 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:54,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:54,977 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:54,977 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:54,977 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:54,977 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:54,977 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:57:54,977 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:57:54,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:54,977 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:54,978 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:54,978 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:54,978 DEBUG [Thread-2783] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:57:54,978 DEBUG [Thread-2783] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:57:54,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:54,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:54,981 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=297 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/B/31934ad7b1e5408fa25560c0cc70ae67 2023-03-21 11:57:55,078 DEBUG [Thread-2783] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:55,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:55,079 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:57:55,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:57:55,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:57:55,080 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:57:55,080 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:55,080 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:57:55,080 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:57:55,081 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:55,081 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:55,081 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:55,081 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:55,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:55,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:55,082 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:55,082 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:57:55,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:55,082 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:55,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:57:55,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:55,082 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:57:55,082 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:57:55,083 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:57:55,083 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:57:55,083 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:55,083 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:57:55,083 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:57:55,083 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:57:55,083 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:55,083 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. started... 2023-03-21 11:57:55,083 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. as already flushing 2023-03-21 11:57:55,083 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:55,083 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:57:55,083 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:57:55,083 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:55,083 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:57:55,083 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:57:55,084 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:55,084 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:55,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:55,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:55,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:55,084 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:55,084 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:57:55,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:55,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:55,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:55,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:55,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:55,085 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:57:55,085 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56155ada[Count = 0] remaining members to acquire global barrier 2023-03-21 11:57:55,085 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:57:55,086 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:55,086 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:55,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:55,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:55,086 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:55,086 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:57:55,086 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:57:55,086 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:57:55,086 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:57:55,087 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:55,087 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:57:55,087 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:55,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:55,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:55,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:55,087 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:57:55,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:55,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:55,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:55,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:55,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:55,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:55,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:55,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:57:55,089 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:57:55,089 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:57:55,089 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:57:55,089 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:57:55,089 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:57:55,089 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:57:55,090 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:55,090 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:55,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:55,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:55,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:55,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:55,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:55,090 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:55,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:55,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:55,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:55,090 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:55,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:55,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:55,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:55,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:55,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:55,091 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:55,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:55,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:55,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:55,092 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:55,092 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:55,092 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:55,092 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:55,092 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:55,092 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:55,092 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:57:55,092 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:57:55,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:55,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:55,092 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:55,093 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:55,093 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:55,093 DEBUG [Thread-2783] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:57:55,093 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:55,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:55,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:55,093 DEBUG [Thread-2783] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:57:55,193 DEBUG [Thread-2783] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:55,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:55,195 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:57:55,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:57:55,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:57:55,196 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:57:55,196 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:55,196 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:57:55,196 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:57:55,196 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:55,196 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:55,197 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:55,197 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:55,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:55,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:55,197 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:55,197 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:57:55,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:55,197 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:55,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:57:55,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:55,198 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:57:55,198 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:57:55,198 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:57:55,198 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:57:55,198 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:55,198 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:57:55,199 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:57:55,199 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:57:55,199 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:55,199 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. started... 2023-03-21 11:57:55,199 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. as already flushing 2023-03-21 11:57:55,199 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:55,199 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:57:55,199 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:57:55,199 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:55,199 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:57:55,199 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:57:55,200 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:55,200 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:55,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:55,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:55,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:55,200 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:55,200 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:57:55,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:55,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:55,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:55,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:55,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:55,201 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:57:55,201 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a78612c[Count = 0] remaining members to acquire global barrier 2023-03-21 11:57:55,201 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:57:55,201 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:55,202 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:55,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:55,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:55,202 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:57:55,202 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:57:55,202 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:57:55,202 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:55,202 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:57:55,202 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:57:55,202 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:55,202 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:55,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:55,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:55,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:55,203 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:57:55,203 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:55,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:55,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48362 deadline: 1679399935199, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:55,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:55,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:55,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:55,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:55,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:55,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:55,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:57:55,205 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:57:55,205 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:57:55,205 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:57:55,205 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:57:55,205 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:57:55,205 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:57:55,206 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:55,206 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:55,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:55,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:55,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:55,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:55,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:55,206 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:55,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:55,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:55,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:55,206 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:55,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:55,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:55,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:55,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:55,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:55,207 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:55,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:55,207 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:55,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48284 deadline: 1679399935203, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:55,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:55,208 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:55,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48302 deadline: 1679399935204, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:55,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:55,208 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:55,208 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:55,208 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:55,208 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:55,208 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:55,208 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:57:55,209 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:57:55,208 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:55,209 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:55,209 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:55,209 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:55,209 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:55,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:55,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:55,209 DEBUG [Thread-2783] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:57:55,209 DEBUG [Thread-2783] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:57:55,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:55,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:55,309 DEBUG [Thread-2783] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:55,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:55,311 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:57:55,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:57:55,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:57:55,311 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:57:55,311 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:55,311 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:57:55,312 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:57:55,312 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:55,312 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:55,312 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:55,312 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:55,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:55,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:55,313 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:55,313 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:57:55,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:55,313 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:55,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:57:55,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:55,313 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:57:55,313 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:57:55,313 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:57:55,314 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:57:55,314 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:55,314 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:57:55,314 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:57:55,314 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:55,314 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:57:55,314 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. started... 2023-03-21 11:57:55,314 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. as already flushing 2023-03-21 11:57:55,314 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:55,314 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:57:55,314 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:57:55,314 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:55,314 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:57:55,314 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:57:55,315 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:55,315 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:55,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:55,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:55,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:55,315 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:55,316 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:57:55,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:55,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:55,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:55,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:55,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:55,317 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:57:55,317 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3bbb6de6[Count = 0] remaining members to acquire global barrier 2023-03-21 11:57:55,317 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:57:55,317 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:55,318 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:55,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:55,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:55,318 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:57:55,318 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:55,318 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:57:55,318 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:57:55,318 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:57:55,319 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:55,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:55,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:55,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:55,319 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:57:55,319 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:55,319 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:57:55,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:55,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:55,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:55,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:55,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:55,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:55,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:55,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:57:55,321 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:57:55,321 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:57:55,321 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:57:55,321 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:57:55,321 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:57:55,321 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:57:55,322 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:55,322 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:55,322 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:55,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:55,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:55,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:55,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:55,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:55,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:55,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:55,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:55,323 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:55,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:55,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:55,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:55,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:55,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:55,324 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:55,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:55,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:55,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:55,325 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:55,325 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:55,325 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:55,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:55,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:55,325 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:55,325 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:57:55,326 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:57:55,325 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:55,325 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:55,326 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:55,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:55,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:55,326 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:55,326 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:55,326 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:55,326 DEBUG [Thread-2783] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:57:55,326 DEBUG [Thread-2783] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:57:55,389 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=297 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/C/29dbd806d71c4ad8879863662b40cf91 2023-03-21 11:57:55,392 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/A/ebd74e81e764452f8fa946cf71020c40 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/ebd74e81e764452f8fa946cf71020c40 2023-03-21 11:57:55,394 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/ebd74e81e764452f8fa946cf71020c40, entries=200, sequenceid=297, filesize=14.3 K 2023-03-21 11:57:55,395 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/B/31934ad7b1e5408fa25560c0cc70ae67 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/31934ad7b1e5408fa25560c0cc70ae67 2023-03-21 11:57:55,397 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/31934ad7b1e5408fa25560c0cc70ae67, entries=150, sequenceid=297, filesize=11.9 K 2023-03-21 11:57:55,398 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/C/29dbd806d71c4ad8879863662b40cf91 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/29dbd806d71c4ad8879863662b40cf91 2023-03-21 11:57:55,400 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/29dbd806d71c4ad8879863662b40cf91, entries=150, sequenceid=297, filesize=11.9 K 2023-03-21 11:57:55,401 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=87.22 KB/89310 for 8dbd4df2f9769e1bcf08d64a8aceb9ad in 835ms, sequenceid=297, compaction requested=true 2023-03-21 11:57:55,401 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8dbd4df2f9769e1bcf08d64a8aceb9ad: 2023-03-21 11:57:55,401 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-21 11:57:55,401 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-21 11:57:55,401 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-21 11:57:55,401 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-21 11:57:55,401 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-21 11:57:55,402 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39578 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-21 11:57:55,402 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37138 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-21 11:57:55,402 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1912): 8dbd4df2f9769e1bcf08d64a8aceb9ad/A is initiating minor compaction (all files) 2023-03-21 11:57:55,402 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1912): 8dbd4df2f9769e1bcf08d64a8aceb9ad/B is initiating minor compaction (all files) 2023-03-21 11:57:55,402 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8dbd4df2f9769e1bcf08d64a8aceb9ad/A in TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:55,402 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8dbd4df2f9769e1bcf08d64a8aceb9ad/B in TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:55,402 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/a0ce75904f18473988823ac43e192f8f, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/71ea227e90e74850a2392ce48309483a, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/ebd74e81e764452f8fa946cf71020c40] into tmpdir=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp, totalSize=38.7 K 2023-03-21 11:57:55,402 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/8d345ea0ff1c4b819d11944cd5014728, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/1a3eddc851604db9b8524ea5b849a2f6, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/31934ad7b1e5408fa25560c0cc70ae67] into tmpdir=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp, totalSize=36.3 K 2023-03-21 11:57:55,402 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting a0ce75904f18473988823ac43e192f8f, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=261, earliestPutTs=1679399873277 2023-03-21 11:57:55,402 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 8d345ea0ff1c4b819d11944cd5014728, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=261, earliestPutTs=1679399873277 2023-03-21 11:57:55,402 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 71ea227e90e74850a2392ce48309483a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=277, earliestPutTs=1679399873601 2023-03-21 11:57:55,402 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 1a3eddc851604db9b8524ea5b849a2f6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=277, earliestPutTs=1679399873601 2023-03-21 11:57:55,403 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting ebd74e81e764452f8fa946cf71020c40, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=297, earliestPutTs=1679399874238 2023-03-21 11:57:55,403 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 31934ad7b1e5408fa25560c0cc70ae67, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=297, earliestPutTs=1679399874238 2023-03-21 11:57:55,407 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] throttle.PressureAwareThroughputController(145): 8dbd4df2f9769e1bcf08d64a8aceb9ad#B#compaction#406 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 57.69 MB/second 2023-03-21 11:57:55,407 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8dbd4df2f9769e1bcf08d64a8aceb9ad#A#compaction#405 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 57.69 MB/second 2023-03-21 11:57:55,426 DEBUG [Thread-2783] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:55,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:55,428 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:57:55,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:57:55,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:57:55,428 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:57:55,428 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:55,428 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:57:55,429 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:57:55,429 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:55,429 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:55,429 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:55,429 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:55,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:55,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:55,430 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:55,430 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:57:55,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:55,430 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:55,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:57:55,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:55,430 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:57:55,430 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:57:55,430 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:57:55,431 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:57:55,431 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:55,431 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:57:55,431 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:57:55,431 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:57:55,431 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:55,431 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. started... 2023-03-21 11:57:55,431 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8dbd4df2f9769e1bcf08d64a8aceb9ad 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-03-21 11:57:55,431 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dbd4df2f9769e1bcf08d64a8aceb9ad, store=A 2023-03-21 11:57:55,431 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:57:55,431 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dbd4df2f9769e1bcf08d64a8aceb9ad, store=B 2023-03-21 11:57:55,431 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:57:55,432 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dbd4df2f9769e1bcf08d64a8aceb9ad, store=C 2023-03-21 11:57:55,432 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:57:55,441 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=313 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/A/04ab5fe013db4e7db0f76dca58215114 2023-03-21 11:57:55,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(9158): Flush requested on 8dbd4df2f9769e1bcf08d64a8aceb9ad 2023-03-21 11:57:55,708 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. as already flushing 2023-03-21 11:57:55,735 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:55,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48362 deadline: 1679399935729, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:55,735 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:55,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48284 deadline: 1679399935729, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:55,735 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:55,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48302 deadline: 1679399935730, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:55,814 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/A/cfa8be38c48a42ab8840f75e842687d2 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/cfa8be38c48a42ab8840f75e842687d2 2023-03-21 11:57:55,815 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/B/aedadcc0d1ea4749bddf13d90362a9b5 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/aedadcc0d1ea4749bddf13d90362a9b5 2023-03-21 11:57:55,820 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8dbd4df2f9769e1bcf08d64a8aceb9ad/A of 8dbd4df2f9769e1bcf08d64a8aceb9ad into cfa8be38c48a42ab8840f75e842687d2(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-21 11:57:55,820 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8dbd4df2f9769e1bcf08d64a8aceb9ad: 2023-03-21 11:57:55,820 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad., storeName=8dbd4df2f9769e1bcf08d64a8aceb9ad/A, priority=13, startTime=1679399875401; duration=0sec 2023-03-21 11:57:55,820 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-21 11:57:55,820 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8dbd4df2f9769e1bcf08d64a8aceb9ad/B of 8dbd4df2f9769e1bcf08d64a8aceb9ad into aedadcc0d1ea4749bddf13d90362a9b5(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-21 11:57:55,820 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-21 11:57:55,820 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8dbd4df2f9769e1bcf08d64a8aceb9ad: 2023-03-21 11:57:55,820 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad., storeName=8dbd4df2f9769e1bcf08d64a8aceb9ad/B, priority=13, startTime=1679399875401; duration=0sec 2023-03-21 11:57:55,820 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-21 11:57:55,821 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37138 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-21 11:57:55,821 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1912): 8dbd4df2f9769e1bcf08d64a8aceb9ad/C is initiating minor compaction (all files) 2023-03-21 11:57:55,821 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8dbd4df2f9769e1bcf08d64a8aceb9ad/C in TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:55,821 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/8e3e6a1fac1d44eaa56ce057b6fbffc9, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/54769945dc7f459d9c2ad0de3690f9e4, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/29dbd806d71c4ad8879863662b40cf91] into tmpdir=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp, totalSize=36.3 K 2023-03-21 11:57:55,821 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 8e3e6a1fac1d44eaa56ce057b6fbffc9, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=261, earliestPutTs=1679399873277 2023-03-21 11:57:55,822 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 54769945dc7f459d9c2ad0de3690f9e4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=277, earliestPutTs=1679399873601 2023-03-21 11:57:55,822 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 29dbd806d71c4ad8879863662b40cf91, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=297, earliestPutTs=1679399874238 2023-03-21 11:57:55,826 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8dbd4df2f9769e1bcf08d64a8aceb9ad#C#compaction#409 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 57.69 MB/second 2023-03-21 11:57:55,841 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:55,841 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:55,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48362 deadline: 1679399935836, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:55,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48284 deadline: 1679399935836, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:55,841 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:55,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48302 deadline: 1679399935836, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:55,848 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=313 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/B/e4a5d2e9c3644955a3f9e0106d147a4c 2023-03-21 11:57:56,045 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:56,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48362 deadline: 1679399936042, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:56,045 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:56,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48284 deadline: 1679399936043, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:56,045 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:56,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48302 deadline: 1679399936043, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:56,243 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/C/4d5963850eb840a0996bfe057f5ca4f2 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/4d5963850eb840a0996bfe057f5ca4f2 2023-03-21 11:57:56,246 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8dbd4df2f9769e1bcf08d64a8aceb9ad/C of 8dbd4df2f9769e1bcf08d64a8aceb9ad into 4d5963850eb840a0996bfe057f5ca4f2(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-21 11:57:56,246 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8dbd4df2f9769e1bcf08d64a8aceb9ad: 2023-03-21 11:57:56,246 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad., storeName=8dbd4df2f9769e1bcf08d64a8aceb9ad/C, priority=13, startTime=1679399875401; duration=0sec 2023-03-21 11:57:56,246 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-21 11:57:56,256 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=313 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/C/6ff5db78cc884b948cf4c91a58b5c3bb 2023-03-21 11:57:56,259 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/A/04ab5fe013db4e7db0f76dca58215114 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/04ab5fe013db4e7db0f76dca58215114 2023-03-21 11:57:56,262 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/04ab5fe013db4e7db0f76dca58215114, entries=150, sequenceid=313, filesize=11.9 K 2023-03-21 11:57:56,262 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/B/e4a5d2e9c3644955a3f9e0106d147a4c as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/e4a5d2e9c3644955a3f9e0106d147a4c 2023-03-21 11:57:56,265 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/e4a5d2e9c3644955a3f9e0106d147a4c, entries=150, sequenceid=313, filesize=11.9 K 2023-03-21 11:57:56,266 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/C/6ff5db78cc884b948cf4c91a58b5c3bb as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/6ff5db78cc884b948cf4c91a58b5c3bb 2023-03-21 11:57:56,271 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/6ff5db78cc884b948cf4c91a58b5c3bb, entries=150, sequenceid=313, filesize=11.9 K 2023-03-21 11:57:56,272 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 8dbd4df2f9769e1bcf08d64a8aceb9ad in 841ms, sequenceid=313, compaction requested=false 2023-03-21 11:57:56,272 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8dbd4df2f9769e1bcf08d64a8aceb9ad: 2023-03-21 11:57:56,272 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:56,272 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:57:56,272 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:57:56,272 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:56,272 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:57:56,272 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:57:56,273 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:56,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:56,273 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:56,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:56,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:56,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:56,273 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:56,273 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:57:56,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:56,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:56,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:56,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:56,274 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:57:56,274 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@691e1750[Count = 0] remaining members to acquire global barrier 2023-03-21 11:57:56,274 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:57:56,275 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:56,275 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:56,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:56,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:56,275 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:57:56,275 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:57:56,275 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:57:56,275 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:56,275 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:57:56,276 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:56,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:56,276 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:57:56,276 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:56,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:56,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:56,276 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:57:56,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:56,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:56,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:56,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:56,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:56,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:56,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:56,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:57:56,278 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:57:56,278 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:57:56,278 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:57:56,278 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:57:56,278 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:57:56,278 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:57:56,279 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:56,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:56,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:56,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:56,279 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:56,279 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:56,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:56,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:56,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:56,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:56,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:56,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:56,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:56,281 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:56,281 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:56,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:56,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:56,282 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:56,282 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:56,282 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:56,282 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:56,283 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:56,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:56,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:56,283 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:56,283 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:56,283 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:57:56,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:56,283 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:57:56,283 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:56,283 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:56,283 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:56,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:56,283 DEBUG [Thread-2783] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:57:56,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:56,283 DEBUG [Thread-2783] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:57:56,283 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:57:56,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(9158): Flush requested on 8dbd4df2f9769e1bcf08d64a8aceb9ad 2023-03-21 11:57:56,350 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8dbd4df2f9769e1bcf08d64a8aceb9ad 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-03-21 11:57:56,351 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dbd4df2f9769e1bcf08d64a8aceb9ad, store=A 2023-03-21 11:57:56,351 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:57:56,351 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dbd4df2f9769e1bcf08d64a8aceb9ad, store=B 2023-03-21 11:57:56,351 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:57:56,351 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dbd4df2f9769e1bcf08d64a8aceb9ad, store=C 2023-03-21 11:57:56,351 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:57:56,383 DEBUG [Thread-2783] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:56,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:56,385 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:56,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48284 deadline: 1679399936380, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:56,385 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:56,385 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:56,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 246 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48362 deadline: 1679399936380, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:56,385 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:57:56,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48302 deadline: 1679399936380, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:56,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:57:56,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:57:56,386 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:57:56,386 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:56,386 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:57:56,386 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:57:56,386 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:56,386 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:56,387 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:56,387 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:56,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:56,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:56,387 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:56,387 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:57:56,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:56,387 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:56,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:57:56,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:56,388 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:57:56,388 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:57:56,388 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:57:56,388 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:57:56,388 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:56,388 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:57:56,388 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:57:56,388 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:57:56,388 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:56,388 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. started... 2023-03-21 11:57:56,388 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. as already flushing 2023-03-21 11:57:56,388 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:56,388 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:57:56,388 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:57:56,388 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:56,389 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:57:56,389 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:57:56,389 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:56,389 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:56,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:56,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:56,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:56,389 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:56,389 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:57:56,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:56,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:56,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:56,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:56,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:56,390 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:57:56,390 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@15215323[Count = 0] remaining members to acquire global barrier 2023-03-21 11:57:56,390 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:57:56,391 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:56,391 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:56,391 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:57:56,391 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:56,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:56,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:56,392 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:57:56,392 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:57:56,392 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:57:56,392 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:56,392 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:57:56,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:56,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:56,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:56,392 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:56,392 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:57:56,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:56,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:56,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:56,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:56,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:56,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:56,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:56,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:57:56,394 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:57:56,394 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:57:56,394 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:57:56,394 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:57:56,394 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:57:56,394 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:57:56,395 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:56,395 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:56,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:56,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:56,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:56,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:56,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:56,395 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:56,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:56,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:56,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:56,395 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:56,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:56,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:56,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:56,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:56,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:56,396 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:56,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:56,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:56,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:56,397 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:56,397 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:56,397 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:56,397 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:56,398 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:56,398 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:56,398 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:56,398 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:56,398 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:57:56,398 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:57:56,398 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:56,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:56,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:56,398 DEBUG [Thread-2783] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:57:56,398 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:56,398 DEBUG [Thread-2783] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:57:56,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:56,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:56,489 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:56,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48284 deadline: 1679399936486, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:56,490 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:56,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 248 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48362 deadline: 1679399936486, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:56,490 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:56,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48302 deadline: 1679399936486, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:56,498 DEBUG [Thread-2783] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:56,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:56,500 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:57:56,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:57:56,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:57:56,500 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:57:56,501 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:56,501 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:57:56,501 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:57:56,501 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:56,501 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:56,501 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:56,501 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:56,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:56,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:56,502 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:56,502 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:57:56,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:56,502 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:56,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:57:56,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:56,502 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:57:56,503 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:57:56,503 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:57:56,503 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:57:56,503 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:56,503 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:57:56,503 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:57:56,503 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:57:56,503 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:56,503 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. started... 2023-03-21 11:57:56,503 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. as already flushing 2023-03-21 11:57:56,503 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:56,503 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:57:56,503 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:57:56,503 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:56,503 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:57:56,503 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:57:56,504 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:56,504 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:56,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:56,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:56,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:56,504 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:56,504 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:57:56,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:56,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:56,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:56,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:56,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:56,505 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:57:56,505 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3cf931aa[Count = 0] remaining members to acquire global barrier 2023-03-21 11:57:56,505 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:57:56,505 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:56,506 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:56,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:56,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:56,506 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:57:56,506 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:57:56,506 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:56,506 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:57:56,506 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:57:56,507 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:57:56,507 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:56,507 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:56,507 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:57:56,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:56,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:56,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:56,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:56,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:56,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:56,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:56,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:56,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:56,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:56,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:57:56,508 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:57:56,508 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:57:56,508 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:57:56,509 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:57:56,509 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:57:56,509 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:57:56,509 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:56,509 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:56,509 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:56,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:56,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:56,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:56,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:56,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:56,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:56,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:56,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:56,510 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:56,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:56,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:56,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:56,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:56,511 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:56,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:56,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:56,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:56,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:56,512 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:56,512 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:56,512 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:56,512 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:56,512 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:56,512 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:56,512 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:57:56,513 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:57:56,512 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:56,512 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:56,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:56,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:56,513 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:56,513 DEBUG [Thread-2783] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:57:56,513 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:57:56,513 DEBUG [Thread-2783] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:57:56,513 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:56,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:56,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:56,613 DEBUG [Thread-2783] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:56,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:56,614 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:57:56,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:57:56,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:57:56,615 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:57:56,615 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:56,615 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:57:56,616 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:57:56,616 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:56,616 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:56,616 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:56,616 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:56,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:56,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:56,617 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:56,617 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:57:56,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:56,617 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:56,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:57:56,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:56,617 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:57:56,617 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:57:56,618 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:57:56,618 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:57:56,618 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:56,618 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:57:56,618 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:57:56,618 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:57:56,618 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:56,618 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. started... 2023-03-21 11:57:56,618 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. as already flushing 2023-03-21 11:57:56,618 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:56,618 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:57:56,618 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:57:56,618 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:56,618 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:57:56,618 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:57:56,619 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:56,619 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:56,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:56,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:56,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:56,619 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:56,619 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:57:56,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:56,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:56,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:56,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:56,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:56,620 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:57:56,620 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ccdbf8e[Count = 0] remaining members to acquire global barrier 2023-03-21 11:57:56,620 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:57:56,620 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:56,621 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:56,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:56,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:56,621 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:57:56,621 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:57:56,621 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:56,621 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:57:56,621 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:57:56,622 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:56,622 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:57:56,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:56,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:56,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:56,622 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:56,622 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:57:56,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:56,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:56,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:56,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:56,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:56,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:56,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:56,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:57:56,623 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:57:56,623 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:57:56,623 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:57:56,623 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:57:56,623 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:57:56,623 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:57:56,624 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:56,624 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:56,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:56,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:56,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:56,624 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:56,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:56,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:56,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:56,625 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:56,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:56,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:56,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:56,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:56,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:56,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:56,625 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:56,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:56,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:56,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:56,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:56,627 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:56,627 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:56,627 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:56,627 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:56,627 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:56,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:56,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:56,627 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:56,627 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:56,627 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:56,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:56,627 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:56,627 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:57:56,627 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:57:56,627 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:56,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:56,627 DEBUG [Thread-2783] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:57:56,627 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:57:56,628 DEBUG [Thread-2783] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:57:56,694 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:56,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48284 deadline: 1679399936691, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:56,694 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:56,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 250 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48362 deadline: 1679399936691, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:56,695 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:56,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48302 deadline: 1679399936691, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:56,728 DEBUG [Thread-2783] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:56,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:56,729 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:57:56,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:57:56,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:57:56,730 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:57:56,730 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:56,730 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:57:56,730 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:57:56,730 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:56,730 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:56,731 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:56,731 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:56,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:56,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:56,731 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:56,731 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:57:56,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:56,732 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:56,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:57:56,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:56,732 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:57:56,732 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:57:56,732 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:57:56,732 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:57:56,732 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:56,732 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:57:56,732 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:57:56,732 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:57:56,732 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:56,732 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. started... 2023-03-21 11:57:56,732 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. as already flushing 2023-03-21 11:57:56,732 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:56,732 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:57:56,732 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:57:56,733 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:56,733 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:57:56,733 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:57:56,733 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:56,733 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:56,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:56,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:56,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:56,733 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:56,734 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:57:56,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:56,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:56,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:56,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:56,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:56,735 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:57:56,735 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@45f097d0[Count = 0] remaining members to acquire global barrier 2023-03-21 11:57:56,735 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:57:56,735 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:56,735 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:56,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:56,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:56,735 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:57:56,735 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:56,735 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:57:56,735 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:57:56,735 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:57:56,736 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:56,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:56,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:56,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:56,736 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:57:56,736 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:56,736 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:57:56,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:56,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:56,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:56,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:56,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:56,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:56,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:56,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:57:56,738 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:57:56,738 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:57:56,738 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:57:56,738 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:57:56,738 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:57:56,738 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:57:56,739 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:56,739 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:56,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:56,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:56,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:56,739 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:56,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:56,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:56,739 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:56,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:56,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:56,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:56,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:56,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:56,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:56,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:56,740 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:56,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:56,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:56,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:56,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:56,741 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:56,741 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:56,741 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:56,741 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:56,741 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:57:56,742 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:57:56,741 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:56,742 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:56,741 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:56,742 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:56,742 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:56,742 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:56,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:56,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:56,742 DEBUG [Thread-2783] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:57:56,742 DEBUG [Thread-2783] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:57:56,742 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:57:56,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:56,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:56,761 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=334 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/A/512dd7bccfc94ab58c1730d5deb22642 2023-03-21 11:57:56,842 DEBUG [Thread-2783] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:56,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:56,844 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:57:56,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:57:56,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:57:56,845 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:57:56,845 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:56,845 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:57:56,845 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:57:56,845 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:56,845 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:56,846 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:56,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:56,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:56,846 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:56,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:56,846 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:56,846 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:57:56,846 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:56,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:57:56,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:56,846 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:57:56,847 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:57:56,847 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:57:56,847 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:57:56,847 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:56,847 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:57:56,847 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:57:56,847 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:57:56,847 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:56,847 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. started... 2023-03-21 11:57:56,847 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. as already flushing 2023-03-21 11:57:56,847 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:56,847 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:57:56,847 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:57:56,847 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:56,847 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:57:56,847 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:57:56,848 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:56,848 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:56,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:56,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:56,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:56,848 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:56,848 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:57:56,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:56,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:56,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:56,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:56,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:56,849 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:57:56,849 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ad76c06[Count = 0] remaining members to acquire global barrier 2023-03-21 11:57:56,849 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:57:56,849 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:56,850 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:56,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:56,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:56,850 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:57:56,850 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:57:56,850 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:57:56,850 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:56,850 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:57:56,851 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:56,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:56,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:56,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:56,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:56,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:56,851 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:57:56,851 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:56,851 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:57:56,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:56,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:56,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:56,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:56,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:56,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:57:56,853 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:57:56,853 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:57:56,853 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:57:56,853 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:57:56,853 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:57:56,853 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:57:56,854 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:56,854 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:56,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:56,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:56,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:56,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:56,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:56,854 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:56,855 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:56,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:56,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:56,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:56,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:56,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:56,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:56,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:56,856 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:56,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:56,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:56,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:56,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:56,858 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:56,858 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:56,858 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:56,858 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:57:56,858 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:57:56,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:56,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:56,858 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:56,859 DEBUG [Thread-2783] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:57:56,859 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:56,858 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:57:56,859 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:56,859 DEBUG [Thread-2783] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:57:56,859 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:56,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:56,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:56,859 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:56,859 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:56,859 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:56,959 DEBUG [Thread-2783] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:56,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:56,961 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:57:56,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:57:56,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:57:56,961 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:57:56,961 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:56,961 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:57:56,961 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:57:56,962 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:56,962 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:56,962 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:56,962 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:56,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:56,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:56,963 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:56,963 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:57:56,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:56,963 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:56,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:57:56,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:56,963 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:57:56,963 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:57:56,963 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:57:56,963 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:57:56,963 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:56,963 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:57:56,963 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:57:56,964 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:57:56,964 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:56,964 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. started... 2023-03-21 11:57:56,964 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. as already flushing 2023-03-21 11:57:56,964 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:56,964 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:57:56,964 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:57:56,964 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:56,964 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:57:56,964 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:57:56,964 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:56,964 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:56,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:56,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:56,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:56,965 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:56,965 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:57:56,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:56,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:56,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:56,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:56,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:56,966 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:57:56,966 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f566297[Count = 0] remaining members to acquire global barrier 2023-03-21 11:57:56,966 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:57:56,966 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:56,966 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:56,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:56,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:56,966 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:57:56,966 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:57:56,966 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:57:56,966 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:56,966 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:57:56,967 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:56,967 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:57:56,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:56,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:56,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:56,967 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:56,967 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:57:56,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:56,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:56,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:56,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:56,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:56,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:56,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:56,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:57:56,969 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:57:56,969 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:57:56,969 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:57:56,969 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:57:56,969 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:57:56,969 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:57:56,969 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:56,969 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:56,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:56,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:56,969 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:56,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:56,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:56,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:56,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:56,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:56,970 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:56,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:56,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:56,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:56,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:56,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:56,970 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:56,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:56,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:56,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:56,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:56,972 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:56,972 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:56,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:56,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:56,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:56,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:56,972 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:56,972 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:56,972 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:56,972 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:56,972 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:57:56,972 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:57:56,972 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:56,972 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:56,972 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:56,972 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:56,972 DEBUG [Thread-2783] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:57:56,972 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:57:56,972 DEBUG [Thread-2783] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:57:57,000 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:57,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48284 deadline: 1679399936995, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:57,000 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:57,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 252 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48362 deadline: 1679399936995, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:57,004 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:57,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48302 deadline: 1679399936997, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:57,073 DEBUG [Thread-2783] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:57,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:57,074 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:57:57,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:57:57,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:57:57,075 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:57:57,075 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:57,075 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:57:57,075 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:57:57,075 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:57,075 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:57,076 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:57,076 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:57,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:57,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:57,076 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:57,076 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:57:57,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:57,076 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:57,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:57:57,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:57,077 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:57:57,077 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:57:57,077 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:57:57,077 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:57:57,077 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:57,077 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:57:57,077 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:57:57,077 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:57:57,077 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:57,077 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. started... 2023-03-21 11:57:57,077 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. as already flushing 2023-03-21 11:57:57,077 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:57,077 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:57:57,077 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:57:57,078 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:57,078 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:57:57,078 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:57:57,078 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:57,078 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:57,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:57,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:57,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:57,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:57,079 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:57,079 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:57:57,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:57,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:57,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:57,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:57,080 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:57:57,080 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1ed6993c[Count = 0] remaining members to acquire global barrier 2023-03-21 11:57:57,080 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:57:57,080 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:57,085 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:57,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:57,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:57,086 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:57:57,086 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:57:57,086 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:57:57,086 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:57,086 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:57:57,086 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:57,086 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:57:57,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:57,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:57,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:57,086 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:57,087 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:57:57,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:57,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:57,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:57,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:57,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:57,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:57,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:57,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:57:57,088 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:57:57,088 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:57:57,088 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:57:57,088 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:57:57,088 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:57:57,088 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:57:57,089 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:57,089 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:57,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:57,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:57,089 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:57,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:57,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:57,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:57,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:57,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:57,089 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:57,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:57,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:57,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:57,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:57,090 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:57,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:57,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:57,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:57,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:57,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:57,092 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:57,092 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:57,092 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:57,092 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:57:57,092 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:57:57,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:57,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:57,092 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:57,092 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:57,092 DEBUG [Thread-2783] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:57:57,092 DEBUG [Thread-2783] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:57:57,092 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:57,092 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:57:57,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:57,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:57,092 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:57,092 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:57,092 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:57,092 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:57,169 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=334 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/B/2da5692354da4dbba2be59bdaa1410a7 2023-03-21 11:57:57,192 DEBUG [Thread-2783] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:57,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:57,194 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:57:57,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:57:57,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:57:57,195 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:57:57,195 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:57,195 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:57:57,195 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:57:57,195 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:57,195 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:57,196 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:57,196 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:57,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:57,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:57,196 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:57,196 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:57:57,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:57,196 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:57,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:57:57,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:57,197 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:57:57,197 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:57:57,197 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:57:57,197 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:57:57,197 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:57,197 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:57:57,197 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:57:57,197 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:57:57,197 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:57,197 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. started... 2023-03-21 11:57:57,197 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. as already flushing 2023-03-21 11:57:57,197 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:57,197 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:57:57,197 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:57:57,197 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:57,197 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:57:57,197 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:57:57,198 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:57,198 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:57,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:57,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:57,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:57,198 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:57,198 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:57:57,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:57,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:57,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:57,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:57,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:57,199 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:57:57,199 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f59d321[Count = 0] remaining members to acquire global barrier 2023-03-21 11:57:57,199 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:57:57,199 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:57,200 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:57,200 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:57,200 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:57:57,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:57,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:57,201 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:57:57,201 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:57:57,201 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:57:57,201 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:57,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:57,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:57,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:57,201 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:57:57,201 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:57,201 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:57:57,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:57,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:57,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:57,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:57,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:57,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:57,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:57,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:57:57,203 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:57:57,203 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:57:57,203 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:57:57,203 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:57:57,203 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:57:57,203 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:57:57,204 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:57,204 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:57,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:57,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:57,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:57,204 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:57,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:57,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:57,204 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:57,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:57,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:57,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:57,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:57,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:57,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:57,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:57,205 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:57,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:57,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:57,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:57,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:57,207 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:57,207 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:57,207 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:57,207 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:57,207 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:57:57,207 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:57:57,207 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:57:57,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:57,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:57,207 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:57,207 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:57,208 DEBUG [Thread-2783] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:57:57,207 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:57,208 DEBUG [Thread-2783] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:57:57,208 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:57,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:57,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:57,208 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:57,208 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:57,308 DEBUG [Thread-2783] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:57,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:57,310 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:57:57,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:57:57,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:57:57,310 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:57:57,310 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:57,310 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:57:57,310 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:57:57,311 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:57,311 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:57,311 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:57,311 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:57,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:57,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:57,312 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:57,312 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:57:57,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:57,312 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:57,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:57:57,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:57,312 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:57:57,313 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:57:57,313 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:57:57,313 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:57:57,313 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:57,313 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:57:57,313 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:57:57,313 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:57:57,313 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:57,313 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. started... 2023-03-21 11:57:57,313 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. as already flushing 2023-03-21 11:57:57,313 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:57,313 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:57:57,313 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:57:57,313 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:57,313 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:57:57,313 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:57:57,314 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:57,314 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:57,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:57,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:57,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:57,314 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:57,314 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:57:57,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:57,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:57,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:57,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:57,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:57,315 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:57:57,315 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:57:57,315 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e8f3390[Count = 0] remaining members to acquire global barrier 2023-03-21 11:57:57,315 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:57,316 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:57,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:57,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:57,316 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:57:57,316 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:57:57,316 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:57:57,316 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:57,316 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:57:57,316 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:57,316 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:57:57,317 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:57,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:57,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:57,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:57,317 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:57:57,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:57,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:57,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:57,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:57,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:57,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:57,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:57,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:57:57,318 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:57:57,318 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:57:57,318 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:57:57,318 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:57:57,318 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:57:57,319 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:57:57,319 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:57,319 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:57,319 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:57,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:57,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:57,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:57,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:57,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:57,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:57,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:57,320 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:57,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:57,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:57,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:57,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:57,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:57,321 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:57,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:57,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:57,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:57,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:57,322 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:57,322 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:57,322 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:57,322 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:57,322 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:57:57,322 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:57:57,322 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:57:57,322 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:57,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:57,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:57,322 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:57,322 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:57,322 DEBUG [Thread-2783] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:57:57,322 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:57,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:57,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:57,322 DEBUG [Thread-2783] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:57:57,323 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:57,323 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:57,423 DEBUG [Thread-2783] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:57,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:57,424 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:57:57,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:57:57,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:57:57,425 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:57:57,425 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:57,425 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:57:57,425 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:57:57,426 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:57,426 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:57,426 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:57,426 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:57,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:57,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:57,426 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:57,426 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:57:57,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:57,427 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:57,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:57:57,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:57,427 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:57:57,427 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:57:57,428 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:57:57,428 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:57:57,428 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:57,428 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:57:57,428 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:57:57,428 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:57:57,428 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:57,428 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. started... 2023-03-21 11:57:57,428 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. as already flushing 2023-03-21 11:57:57,428 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:57,428 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:57:57,428 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:57:57,428 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:57,428 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:57:57,428 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:57:57,429 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:57,429 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:57,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:57,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:57,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:57,429 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:57,429 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:57:57,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:57,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:57,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:57,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:57,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:57,430 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:57:57,430 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2251ac0c[Count = 0] remaining members to acquire global barrier 2023-03-21 11:57:57,430 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:57:57,430 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:57,431 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:57,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:57,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:57,431 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:57,431 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:57:57,431 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:57:57,431 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:57:57,431 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:57:57,432 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:57,432 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:57:57,432 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:57,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:57,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:57,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:57,432 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:57:57,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:57,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:57,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:57,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:57,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:57,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:57,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:57,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:57:57,434 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:57:57,434 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:57:57,434 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:57:57,434 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:57:57,434 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:57:57,434 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:57:57,434 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:57,434 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:57,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:57,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:57,434 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:57,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:57,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:57,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:57,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:57,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:57,435 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:57,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:57,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:57,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:57,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:57,435 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:57,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:57,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:57,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:57,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:57,437 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:57,437 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:57,437 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:57,437 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:57:57,437 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:57:57,437 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:57,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:57,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:57,437 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:57,437 DEBUG [Thread-2783] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:57:57,437 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:57:57,437 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:57,437 DEBUG [Thread-2783] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:57:57,437 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:57,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:57,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:57,437 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:57,437 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:57,437 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:57,502 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:57,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48284 deadline: 1679399937502, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:57,511 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:57,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 254 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48362 deadline: 1679399937502, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:57,512 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:57,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48302 deadline: 1679399937509, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:57,537 DEBUG [Thread-2783] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:57,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:57,539 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:57:57,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:57:57,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:57:57,540 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:57:57,540 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:57,540 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:57:57,540 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:57:57,540 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:57,540 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:57,541 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:57,541 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:57,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:57,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:57,541 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:57,541 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:57:57,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:57,541 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:57,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:57:57,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:57,542 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:57:57,542 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:57:57,542 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:57:57,542 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:57:57,542 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:57,542 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:57:57,542 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:57:57,542 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:57:57,542 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:57,542 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. started... 2023-03-21 11:57:57,542 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. as already flushing 2023-03-21 11:57:57,542 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:57,542 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:57:57,542 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:57:57,542 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:57,542 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:57:57,542 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:57:57,543 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:57,543 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:57,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:57,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:57,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:57,543 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:57,543 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:57:57,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:57,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:57,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:57,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:57,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:57,544 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:57:57,544 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@864de03[Count = 0] remaining members to acquire global barrier 2023-03-21 11:57:57,544 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:57:57,544 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:57,545 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:57,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:57,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:57,545 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:57:57,545 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:57:57,545 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:57:57,545 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:57,545 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:57:57,546 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:57,546 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:57:57,546 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:57,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:57,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:57,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:57,546 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:57:57,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:57,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:57,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:57,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:57,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:57,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:57,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:57,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:57:57,549 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:57:57,549 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:57:57,549 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:57:57,549 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:57:57,549 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:57:57,549 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:57:57,550 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:57,550 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:57,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:57,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:57,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:57,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:57,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:57,550 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:57,550 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:57,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:57,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:57,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:57,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:57,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:57,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:57,551 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:57,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:57,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:57,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:57,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:57,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:57,554 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:57,554 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:57,554 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:57,554 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:57:57,554 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:57:57,554 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:57,554 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:57,554 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:57,554 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:57,554 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:57,554 DEBUG [Thread-2783] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:57:57,554 DEBUG [Thread-2783] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:57:57,554 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:57:57,554 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:57,554 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:57,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:57,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:57,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:57,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:57,579 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=334 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/C/3aeff156e4274296bae04c4b64f94ac8 2023-03-21 11:57:57,583 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/A/512dd7bccfc94ab58c1730d5deb22642 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/512dd7bccfc94ab58c1730d5deb22642 2023-03-21 11:57:57,585 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/512dd7bccfc94ab58c1730d5deb22642, entries=150, sequenceid=334, filesize=11.9 K 2023-03-21 11:57:57,586 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/B/2da5692354da4dbba2be59bdaa1410a7 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/2da5692354da4dbba2be59bdaa1410a7 2023-03-21 11:57:57,588 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/2da5692354da4dbba2be59bdaa1410a7, entries=150, sequenceid=334, filesize=11.9 K 2023-03-21 11:57:57,589 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/C/3aeff156e4274296bae04c4b64f94ac8 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/3aeff156e4274296bae04c4b64f94ac8 2023-03-21 11:57:57,591 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/3aeff156e4274296bae04c4b64f94ac8, entries=150, sequenceid=334, filesize=11.9 K 2023-03-21 11:57:57,592 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=87.22 KB/89310 for 8dbd4df2f9769e1bcf08d64a8aceb9ad in 1242ms, sequenceid=334, compaction requested=true 2023-03-21 11:57:57,592 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8dbd4df2f9769e1bcf08d64a8aceb9ad: 2023-03-21 11:57:57,592 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-21 11:57:57,592 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-21 11:57:57,592 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-21 11:57:57,592 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-21 11:57:57,592 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-21 11:57:57,593 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37240 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-21 11:57:57,593 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37240 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-21 11:57:57,593 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1912): 8dbd4df2f9769e1bcf08d64a8aceb9ad/A is initiating minor compaction (all files) 2023-03-21 11:57:57,593 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1912): 8dbd4df2f9769e1bcf08d64a8aceb9ad/B is initiating minor compaction (all files) 2023-03-21 11:57:57,593 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8dbd4df2f9769e1bcf08d64a8aceb9ad/A in TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:57,593 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8dbd4df2f9769e1bcf08d64a8aceb9ad/B in TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:57,593 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/cfa8be38c48a42ab8840f75e842687d2, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/04ab5fe013db4e7db0f76dca58215114, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/512dd7bccfc94ab58c1730d5deb22642] into tmpdir=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp, totalSize=36.4 K 2023-03-21 11:57:57,593 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/aedadcc0d1ea4749bddf13d90362a9b5, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/e4a5d2e9c3644955a3f9e0106d147a4c, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/2da5692354da4dbba2be59bdaa1410a7] into tmpdir=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp, totalSize=36.4 K 2023-03-21 11:57:57,593 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting cfa8be38c48a42ab8840f75e842687d2, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=297, earliestPutTs=1679399874238 2023-03-21 11:57:57,594 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting aedadcc0d1ea4749bddf13d90362a9b5, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=297, earliestPutTs=1679399874238 2023-03-21 11:57:57,594 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 04ab5fe013db4e7db0f76dca58215114, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=313, earliestPutTs=1679399874578 2023-03-21 11:57:57,594 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting e4a5d2e9c3644955a3f9e0106d147a4c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=313, earliestPutTs=1679399874578 2023-03-21 11:57:57,594 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 512dd7bccfc94ab58c1730d5deb22642, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=334, earliestPutTs=1679399875727 2023-03-21 11:57:57,594 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 2da5692354da4dbba2be59bdaa1410a7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=334, earliestPutTs=1679399875727 2023-03-21 11:57:57,601 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8dbd4df2f9769e1bcf08d64a8aceb9ad#A#compaction#414 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 57.69 MB/second 2023-03-21 11:57:57,605 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] throttle.PressureAwareThroughputController(145): 8dbd4df2f9769e1bcf08d64a8aceb9ad#B#compaction#415 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 57.69 MB/second 2023-03-21 11:57:57,654 DEBUG [Thread-2783] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:57,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:57,656 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:57:57,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:57:57,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:57:57,657 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:57:57,657 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:57,657 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:57:57,657 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:57:57,657 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:57,657 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:57,658 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:57,658 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:57,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:57,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:57,658 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:57,658 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:57:57,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:57,658 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:57,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:57:57,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:57,658 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:57:57,659 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:57:57,659 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:57:57,659 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:57:57,659 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:57,659 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:57:57,659 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:57:57,659 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:57:57,659 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:57,659 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. started... 2023-03-21 11:57:57,659 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8dbd4df2f9769e1bcf08d64a8aceb9ad 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-03-21 11:57:57,659 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dbd4df2f9769e1bcf08d64a8aceb9ad, store=A 2023-03-21 11:57:57,659 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:57:57,659 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dbd4df2f9769e1bcf08d64a8aceb9ad, store=B 2023-03-21 11:57:57,659 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:57:57,660 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dbd4df2f9769e1bcf08d64a8aceb9ad, store=C 2023-03-21 11:57:57,660 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:57:58,014 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/A/99b529bc783042bbbad5234c5bd1ec62 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/99b529bc783042bbbad5234c5bd1ec62 2023-03-21 11:57:58,018 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/B/6a87bfe9de63438cba29724c00e3692b as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/6a87bfe9de63438cba29724c00e3692b 2023-03-21 11:57:58,020 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8dbd4df2f9769e1bcf08d64a8aceb9ad/A of 8dbd4df2f9769e1bcf08d64a8aceb9ad into 99b529bc783042bbbad5234c5bd1ec62(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-21 11:57:58,020 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8dbd4df2f9769e1bcf08d64a8aceb9ad: 2023-03-21 11:57:58,020 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad., storeName=8dbd4df2f9769e1bcf08d64a8aceb9ad/A, priority=13, startTime=1679399877592; duration=0sec 2023-03-21 11:57:58,020 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-21 11:57:58,020 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-21 11:57:58,022 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37240 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-21 11:57:58,022 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1912): 8dbd4df2f9769e1bcf08d64a8aceb9ad/C is initiating minor compaction (all files) 2023-03-21 11:57:58,022 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8dbd4df2f9769e1bcf08d64a8aceb9ad/C in TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:58,022 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/4d5963850eb840a0996bfe057f5ca4f2, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/6ff5db78cc884b948cf4c91a58b5c3bb, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/3aeff156e4274296bae04c4b64f94ac8] into tmpdir=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp, totalSize=36.4 K 2023-03-21 11:57:58,022 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 4d5963850eb840a0996bfe057f5ca4f2, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=297, earliestPutTs=1679399874238 2023-03-21 11:57:58,023 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 6ff5db78cc884b948cf4c91a58b5c3bb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=313, earliestPutTs=1679399874578 2023-03-21 11:57:58,023 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 3aeff156e4274296bae04c4b64f94ac8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=334, earliestPutTs=1679399875727 2023-03-21 11:57:58,023 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8dbd4df2f9769e1bcf08d64a8aceb9ad/B of 8dbd4df2f9769e1bcf08d64a8aceb9ad into 6a87bfe9de63438cba29724c00e3692b(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-21 11:57:58,023 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8dbd4df2f9769e1bcf08d64a8aceb9ad: 2023-03-21 11:57:58,023 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad., storeName=8dbd4df2f9769e1bcf08d64a8aceb9ad/B, priority=13, startTime=1679399877592; duration=0sec 2023-03-21 11:57:58,023 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-21 11:57:58,027 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8dbd4df2f9769e1bcf08d64a8aceb9ad#C#compaction#417 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 57.69 MB/second 2023-03-21 11:57:58,065 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=350 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/A/5d821ecd508a4491b4a9f7cfe200d067 2023-03-21 11:57:58,075 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=350 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/B/8d8f54ca750f481798cc77d788b87ae0 2023-03-21 11:57:58,082 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=350 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/C/016f5e93911c42df83d3f00d4fad3c86 2023-03-21 11:57:58,085 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/A/5d821ecd508a4491b4a9f7cfe200d067 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/5d821ecd508a4491b4a9f7cfe200d067 2023-03-21 11:57:58,088 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/5d821ecd508a4491b4a9f7cfe200d067, entries=150, sequenceid=350, filesize=11.9 K 2023-03-21 11:57:58,088 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/B/8d8f54ca750f481798cc77d788b87ae0 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/8d8f54ca750f481798cc77d788b87ae0 2023-03-21 11:57:58,091 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/8d8f54ca750f481798cc77d788b87ae0, entries=150, sequenceid=350, filesize=11.9 K 2023-03-21 11:57:58,091 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/C/016f5e93911c42df83d3f00d4fad3c86 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/016f5e93911c42df83d3f00d4fad3c86 2023-03-21 11:57:58,094 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/016f5e93911c42df83d3f00d4fad3c86, entries=150, sequenceid=350, filesize=11.9 K 2023-03-21 11:57:58,094 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=0 B/0 for 8dbd4df2f9769e1bcf08d64a8aceb9ad in 435ms, sequenceid=350, compaction requested=false 2023-03-21 11:57:58,095 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8dbd4df2f9769e1bcf08d64a8aceb9ad: 2023-03-21 11:57:58,095 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:58,095 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:57:58,095 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:57:58,095 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,095 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:57:58,095 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:57:58,096 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:58,096 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:58,096 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:57:58,096 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:58,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:58,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:58,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:58,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:58,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:58,098 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:57:58,098 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1935d4f5[Count = 0] remaining members to acquire global barrier 2023-03-21 11:57:58,098 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:57:58,098 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:58,099 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:58,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:58,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:58,099 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:57:58,099 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:57:58,099 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:57:58,099 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,099 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:57:58,100 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,100 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:57:58,100 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:58,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:58,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:58,100 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:57:58,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:58,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:58,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:58,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:58,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:58,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:57:58,103 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:57:58,103 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:57:58,103 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:57:58,103 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:57:58,103 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:57:58,103 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:57:58,104 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:58,104 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:58,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:58,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:58,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:58,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:58,104 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:58,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:58,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:58,104 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:58,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:58,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:58,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:58,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:58,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:58,106 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:58,107 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,107 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:58,107 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:58,107 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,107 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:58,107 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:58,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:58,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:58,107 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:58,107 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:58,107 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:58,107 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:57:58,108 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:58,108 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:57:58,108 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:57:58,108 DEBUG [Thread-2783] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:57:58,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:58,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:58,108 DEBUG [Thread-2783] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:57:58,208 DEBUG [Thread-2783] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:58,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:58,210 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:57:58,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:57:58,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:57:58,211 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:57:58,211 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:58,211 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:57:58,211 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:57:58,211 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:58,211 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:58,212 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:58,212 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:58,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:58,212 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,212 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:57:58,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:58,216 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:58,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:57:58,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:58,217 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:57:58,217 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:57:58,217 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:57:58,217 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:57:58,217 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:58,217 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:57:58,217 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:57:58,217 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:57:58,217 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:58,217 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. started... 2023-03-21 11:57:58,217 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8dbd4df2f9769e1bcf08d64a8aceb9ad: 2023-03-21 11:57:58,217 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:58,217 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:57:58,217 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:57:58,217 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,217 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:57:58,217 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:57:58,218 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,218 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:58,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:58,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:58,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:58,219 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:58,219 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:57:58,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:58,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:58,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:58,220 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:57:58,220 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e6177f0[Count = 0] remaining members to acquire global barrier 2023-03-21 11:57:58,220 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:57:58,220 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:58,220 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:58,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:58,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:58,221 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:57:58,221 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:57:58,221 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:57:58,221 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,221 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:57:58,222 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:57:58,222 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,222 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:58,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:58,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:58,222 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:57:58,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:58,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:58,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:58,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:58,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:58,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:57:58,224 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:57:58,224 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:57:58,224 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:57:58,224 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:57:58,224 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:57:58,224 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:57:58,225 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:58,225 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:58,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:58,225 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:58,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:58,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:58,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:58,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:58,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:58,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:58,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:58,226 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:58,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:58,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:58,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:58,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,227 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:58,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:58,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,228 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,228 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:58,228 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:58,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:58,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:58,229 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:57:58,229 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:57:58,229 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:58,229 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:57:58,229 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:58,229 DEBUG [Thread-2783] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:57:58,229 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:58,229 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,229 DEBUG [Thread-2783] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:57:58,229 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:58,229 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:58,229 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:58,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:58,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:58,329 DEBUG [Thread-2783] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:58,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:58,331 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:57:58,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:57:58,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:57:58,335 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:57:58,335 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:58,335 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:57:58,335 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:57:58,335 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:58,335 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:58,336 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,336 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:58,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:58,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:58,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:58,336 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,336 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:57:58,336 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:58,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:57:58,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:58,337 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:57:58,337 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:57:58,337 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:57:58,337 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:57:58,337 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:58,337 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:57:58,337 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:57:58,337 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:57:58,337 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:58,337 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. started... 2023-03-21 11:57:58,337 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8dbd4df2f9769e1bcf08d64a8aceb9ad: 2023-03-21 11:57:58,337 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:58,337 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:57:58,337 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:57:58,338 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,338 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:57:58,338 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:57:58,338 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,338 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:58,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:58,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:58,339 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:58,339 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:57:58,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:58,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:58,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:58,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:58,340 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:57:58,340 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@717f81fc[Count = 0] remaining members to acquire global barrier 2023-03-21 11:57:58,340 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:57:58,340 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:58,340 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:58,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:58,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:58,340 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,341 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:57:58,340 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:57:58,341 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:57:58,341 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:57:58,341 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,341 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:57:58,341 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:58,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:58,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:58,341 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:57:58,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:58,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:58,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:58,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:58,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:58,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:57:58,343 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:57:58,343 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:57:58,343 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:57:58,343 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:57:58,343 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:57:58,343 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:57:58,344 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:58,344 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:58,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:58,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:58,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:58,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:58,344 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:58,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:58,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:58,345 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:58,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:58,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:58,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:58,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:58,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:58,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,346 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:58,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:58,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,347 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,348 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:58,348 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:57:58,348 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:57:58,348 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:58,348 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:58,348 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:58,348 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:58,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:58,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:58,348 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,348 DEBUG [Thread-2783] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:57:58,348 DEBUG [Thread-2783] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:57:58,348 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:58,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:58,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:58,348 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:58,348 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:58,437 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/C/381d152397b3409699f3688d0fc8048f as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/381d152397b3409699f3688d0fc8048f 2023-03-21 11:57:58,439 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8dbd4df2f9769e1bcf08d64a8aceb9ad/C of 8dbd4df2f9769e1bcf08d64a8aceb9ad into 381d152397b3409699f3688d0fc8048f(size=12.7 K), total size for store is 24.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-21 11:57:58,440 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8dbd4df2f9769e1bcf08d64a8aceb9ad: 2023-03-21 11:57:58,440 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad., storeName=8dbd4df2f9769e1bcf08d64a8aceb9ad/C, priority=13, startTime=1679399877592; duration=0sec 2023-03-21 11:57:58,440 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-21 11:57:58,448 DEBUG [Thread-2783] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:58,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:58,450 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:57:58,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:57:58,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:57:58,451 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:57:58,451 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:58,451 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:57:58,451 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:57:58,451 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:58,451 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:58,452 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:58,452 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:58,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:58,452 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,452 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:57:58,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:58,452 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:58,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:57:58,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:58,453 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:57:58,453 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:57:58,453 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:57:58,453 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:57:58,453 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:58,453 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:57:58,453 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:57:58,453 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:57:58,454 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:58,454 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. started... 2023-03-21 11:57:58,454 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8dbd4df2f9769e1bcf08d64a8aceb9ad: 2023-03-21 11:57:58,454 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:58,454 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:57:58,454 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:57:58,454 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,454 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:57:58,454 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:57:58,455 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,455 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:58,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:58,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:58,456 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:58,456 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:57:58,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:58,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:58,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:58,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:58,457 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:57:58,457 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f45dd16[Count = 0] remaining members to acquire global barrier 2023-03-21 11:57:58,457 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:57:58,457 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:58,457 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:58,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:58,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:58,457 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:57:58,457 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,457 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:57:58,457 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:57:58,458 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:57:58,459 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,459 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:57:58,459 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:58,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:58,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:58,459 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:57:58,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:58,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:58,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:58,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:58,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:58,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:57:58,461 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:57:58,461 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:57:58,461 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:57:58,461 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:57:58,461 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:57:58,461 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:57:58,462 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:58,462 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:58,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:58,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:58,462 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:58,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:58,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:58,462 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:58,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:58,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:58,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:58,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:58,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:58,463 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:58,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:58,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:58,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:58,464 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,464 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:58,464 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:58,464 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,464 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:58,464 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:58,464 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:58,465 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:58,465 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:58,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:58,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:58,465 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:58,465 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:57:58,465 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:57:58,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:58,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:58,465 DEBUG [Thread-2783] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:57:58,465 DEBUG [Thread-2783] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:57:58,465 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:57:58,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(9158): Flush requested on 8dbd4df2f9769e1bcf08d64a8aceb9ad 2023-03-21 11:57:58,516 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8dbd4df2f9769e1bcf08d64a8aceb9ad 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-21 11:57:58,516 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dbd4df2f9769e1bcf08d64a8aceb9ad, store=A 2023-03-21 11:57:58,516 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:57:58,516 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dbd4df2f9769e1bcf08d64a8aceb9ad, store=B 2023-03-21 11:57:58,516 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:57:58,516 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dbd4df2f9769e1bcf08d64a8aceb9ad, store=C 2023-03-21 11:57:58,516 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:57:58,546 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679399878525, max timestamp=1679399878544Num uniques 859; 2023-03-21 11:57:58,546 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679399878525, max timestamp=1679399878544Num uniques 859; 2023-03-21 11:57:58,546 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-21 11:57:58,546 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-21 11:57:58,546 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679399878525, max timestamp=1679399878544Num uniques 859; 2023-03-21 11:57:58,546 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-21 11:57:58,565 DEBUG [Thread-2783] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:58,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:58,567 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:57:58,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:57:58,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:57:58,568 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:57:58,568 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:58,568 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:57:58,568 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:57:58,568 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:58,568 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:58,569 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:58,569 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:58,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:58,569 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,569 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:57:58,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:58,569 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:58,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:57:58,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:58,570 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:57:58,570 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:57:58,570 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:57:58,570 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:57:58,570 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:58,570 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:57:58,570 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:57:58,570 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:58,570 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. started... 2023-03-21 11:57:58,570 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:57:58,570 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. as already flushing 2023-03-21 11:57:58,570 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:58,570 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:57:58,570 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:57:58,570 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,570 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:57:58,570 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:57:58,571 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:58,571 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:58,572 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:57:58,572 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:58,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:58,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:58,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:58,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:58,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:58,573 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:57:58,573 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@68602877[Count = 0] remaining members to acquire global barrier 2023-03-21 11:57:58,573 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:57:58,573 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:58,574 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:58,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:58,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:58,574 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:57:58,574 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,574 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:57:58,574 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:57:58,574 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:57:58,575 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:57:58,575 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,575 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:58,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:58,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:58,575 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:57:58,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:58,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:58,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:58,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:58,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:58,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:57:58,576 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:57:58,576 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:57:58,576 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:57:58,576 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:57:58,576 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:57:58,576 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:57:58,577 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:58,577 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:58,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:58,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:58,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:58,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:58,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:58,577 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:58,578 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:58,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:58,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:58,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:58,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:58,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:58,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:58,579 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:58,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:58,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,580 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,580 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:58,580 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:58,580 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:58,581 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:58,581 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:58,581 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:58,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:58,581 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:58,581 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:57:58,581 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:57:58,581 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:58,581 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:58,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:58,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:58,581 DEBUG [Thread-2783] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:57:58,581 DEBUG [Thread-2783] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:57:58,581 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:58,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48302 deadline: 1679399938574, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,582 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:57:58,586 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:58,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 268 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48362 deadline: 1679399938582, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,588 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:58,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48284 deadline: 1679399938582, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,681 DEBUG [Thread-2783] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:58,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:58,683 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:57:58,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:57:58,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:57:58,684 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:57:58,684 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:58,684 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:57:58,684 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:57:58,684 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:58,684 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:58,685 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,685 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:58,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:58,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:58,685 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,685 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:57:58,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:58,685 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:58,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:57:58,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:58,686 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:57:58,686 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:57:58,686 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:57:58,686 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:57:58,686 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:58,686 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:57:58,686 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:57:58,686 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:57:58,686 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:58,686 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. started... 2023-03-21 11:57:58,686 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. as already flushing 2023-03-21 11:57:58,686 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:58,686 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:57:58,686 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:57:58,686 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,686 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:57:58,686 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:57:58,687 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:58,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48302 deadline: 1679399938682, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,687 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:58,687 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:58,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:58,687 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:58,687 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:57:58,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:58,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:58,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:58,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:58,688 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:57:58,688 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3deb0c7c[Count = 0] remaining members to acquire global barrier 2023-03-21 11:57:58,688 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:57:58,689 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:58,689 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:58,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:58,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:58,689 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:57:58,689 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:57:58,689 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,689 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:57:58,689 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:57:58,690 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,690 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:57:58,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:58,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:58,690 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:58,690 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:57:58,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:58,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:58,691 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:58,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 270 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48362 deadline: 1679399938687, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:58,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:58,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:58,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:57:58,692 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:57:58,692 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:57:58,692 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:57:58,692 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:57:58,692 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:57:58,692 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:57:58,693 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:58,693 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:58,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:58,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:58,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:58,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:58,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:58,693 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:58,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:58,693 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:58,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:58,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:58,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:58,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:58,694 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:58,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48284 deadline: 1679399938689, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:58,694 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:58,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:58,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,695 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:58,695 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:58,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:58,695 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:58,696 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:58,695 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:58,696 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:57:58,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:58,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:58,696 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:57:58,696 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:58,696 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,696 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:58,696 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:58,696 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:58,696 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:57:58,696 DEBUG [Thread-2783] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:57:58,696 DEBUG [Thread-2783] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:57:58,796 DEBUG [Thread-2783] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:58,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:58,799 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:57:58,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:57:58,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:57:58,800 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:57:58,800 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:58,800 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:57:58,800 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:57:58,800 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:58,800 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:58,801 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,801 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:58,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:58,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:58,801 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,801 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:57:58,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:58,801 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:58,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:57:58,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:58,802 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:57:58,802 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:57:58,802 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:57:58,802 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:57:58,802 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:58,802 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:57:58,802 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:57:58,802 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:57:58,802 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:58,802 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. started... 2023-03-21 11:57:58,802 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. as already flushing 2023-03-21 11:57:58,802 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:58,802 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:57:58,802 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:57:58,802 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,802 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:57:58,802 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:57:58,803 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:58,803 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:58,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:58,803 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:58,803 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:57:58,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:58,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:58,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:58,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:58,804 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:57:58,804 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@30161a5c[Count = 0] remaining members to acquire global barrier 2023-03-21 11:57:58,804 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:57:58,804 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:58,806 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:58,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:58,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:58,806 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:57:58,806 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:57:58,806 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:57:58,806 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,806 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:57:58,806 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,806 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:57:58,806 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:58,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:58,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:58,807 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:57:58,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:58,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:58,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:58,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:58,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:58,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:57:58,808 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:57:58,808 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:57:58,809 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:57:58,809 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:57:58,809 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:57:58,809 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:57:58,809 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:58,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:58,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:58,809 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:58,809 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:58,810 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:58,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:58,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:58,810 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:58,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:58,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:58,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:58,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:58,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:58,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:58,812 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,812 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:58,812 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:57:58,812 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:57:58,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:58,812 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:58,812 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:58,812 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,812 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:58,812 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:58,812 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:58,812 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:58,812 DEBUG [Thread-2783] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:57:58,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:58,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:58,812 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:58,813 DEBUG [Thread-2783] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:57:58,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:58,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:58,813 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:57:58,896 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:58,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48302 deadline: 1679399938888, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,896 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:58,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 272 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48362 deadline: 1679399938892, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,897 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:58,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48284 deadline: 1679399938894, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,913 DEBUG [Thread-2783] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:58,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:58,915 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:57:58,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:57:58,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:57:58,915 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:57:58,915 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:58,915 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:57:58,915 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:57:58,916 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:58,916 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:58,916 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,916 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:58,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:58,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:58,917 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,917 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:57:58,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:58,917 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:58,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:57:58,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:58,917 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:57:58,918 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:57:58,918 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:57:58,918 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:57:58,918 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:58,918 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:57:58,918 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:57:58,918 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:57:58,918 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:58,918 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. started... 2023-03-21 11:57:58,918 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. as already flushing 2023-03-21 11:57:58,918 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:58,918 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:57:58,918 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:57:58,918 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,918 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:57:58,918 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:57:58,919 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,919 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:58,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:58,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:58,919 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:58,919 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:57:58,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:58,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:58,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:58,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:58,920 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:57:58,920 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@414acc25[Count = 0] remaining members to acquire global barrier 2023-03-21 11:57:58,920 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:57:58,921 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:58,921 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:58,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:58,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:58,921 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:57:58,921 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:57:58,921 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,921 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:57:58,921 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:57:58,922 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=366 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/A/a4083b568dad40819a4740c449a3d8e1 2023-03-21 11:57:58,922 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:58,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:58,922 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:57:58,922 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:58,922 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:57:58,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:58,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:58,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:58,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:58,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:58,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:57:58,925 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:57:58,925 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:57:58,925 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:57:58,925 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:57:58,925 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:57:58,925 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:57:58,927 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:58,927 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:58,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:58,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:58,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:58,927 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:58,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:58,927 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:58,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:58,927 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,927 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:58,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:58,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:58,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:58,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:58,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:58,929 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:58,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:58,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,930 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,930 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:58,930 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:58,930 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:58,930 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:58,930 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:58,930 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:58,930 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:57:58,930 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:57:58,930 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:58,930 DEBUG [Thread-2783] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:57:58,930 DEBUG [Thread-2783] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:57:58,931 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:57:58,930 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:58,930 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:58,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:58,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:58,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:58,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:59,031 DEBUG [Thread-2783] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:59,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:59,032 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:57:59,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:57:59,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:57:59,033 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:57:59,033 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:59,033 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:57:59,033 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:57:59,033 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:59,033 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:59,034 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:59,034 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:59,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:59,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:59,034 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:59,034 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:57:59,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:59,035 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:59,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:57:59,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:59,035 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:57:59,037 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:57:59,037 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:57:59,037 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:57:59,037 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:59,037 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:57:59,037 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:57:59,037 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:57:59,037 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:59,037 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. started... 2023-03-21 11:57:59,037 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. as already flushing 2023-03-21 11:57:59,037 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:59,037 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:57:59,037 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:57:59,037 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:59,037 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:57:59,037 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:57:59,038 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:59,038 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:59,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:59,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:59,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:59,038 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:59,038 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:57:59,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:59,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:59,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:59,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:59,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:59,040 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:57:59,040 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@280077eb[Count = 0] remaining members to acquire global barrier 2023-03-21 11:57:59,040 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:57:59,040 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:59,041 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:59,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:59,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:59,042 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:57:59,042 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:57:59,042 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:57:59,042 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:57:59,042 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:59,042 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:57:59,043 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:59,043 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:57:59,043 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:57:59,043 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:57:59,043 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:57:59,043 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:57:59,043 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:57:59,043 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:57:59,044 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:59,044 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:59,044 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:59,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:59,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:59,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:59,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:59,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:59,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:59,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:59,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:59,045 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:59,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:59,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:59,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:59,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:59,046 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:59,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:59,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:59,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:59,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:59,048 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:59,049 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:59,049 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:59,049 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:59,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:59,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:59,049 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:59,049 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:57:59,049 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:57:59,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:59,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:59,049 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:59,049 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:59,049 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:59,049 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:59,049 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:59,049 DEBUG [Thread-2783] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:57:59,049 DEBUG [Thread-2783] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:57:59,149 DEBUG [Thread-2783] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:59,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:59,151 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:57:59,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:57:59,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:57:59,152 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:57:59,152 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:59,152 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:57:59,152 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:57:59,152 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:59,152 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:59,153 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:59,153 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:59,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:59,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:59,153 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:59,153 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:57:59,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:59,153 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:59,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:57:59,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:59,154 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:57:59,154 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:57:59,154 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:57:59,154 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:57:59,154 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:59,154 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:57:59,154 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:57:59,154 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:57:59,154 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:59,154 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. started... 2023-03-21 11:57:59,154 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. as already flushing 2023-03-21 11:57:59,154 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:59,155 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:57:59,155 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:57:59,155 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:59,155 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:57:59,155 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:57:59,156 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:59,156 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:59,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:59,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:59,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:59,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:59,156 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:59,156 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:57:59,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:59,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:59,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:59,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:59,157 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:57:59,157 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4ded167b[Count = 0] remaining members to acquire global barrier 2023-03-21 11:57:59,157 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:57:59,157 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:59,157 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:59,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:59,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:59,158 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:57:59,158 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:57:59,158 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:59,158 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:57:59,158 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:57:59,159 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:59,159 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:57:59,159 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:59,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:59,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:59,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:59,159 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:57:59,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:59,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:59,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:59,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:59,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:59,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:59,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:59,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:57:59,163 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:57:59,163 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:57:59,163 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:57:59,163 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:57:59,163 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:57:59,163 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:57:59,164 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:59,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:59,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:59,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:59,164 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:59,164 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:59,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:59,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:59,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:59,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:59,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:59,164 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:59,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:59,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:59,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:59,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:59,165 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:59,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:59,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:59,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:59,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:59,166 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:59,166 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:59,166 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:59,166 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:59,166 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:59,166 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:59,166 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:57:59,167 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:57:59,166 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:57:59,166 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:59,166 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:59,167 DEBUG [Thread-2783] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:57:59,167 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:59,167 DEBUG [Thread-2783] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:57:59,167 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:59,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:59,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:59,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:59,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:59,201 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:59,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 274 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48362 deadline: 1679399939197, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:59,201 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:59,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48284 deadline: 1679399939199, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:59,201 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:59,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48302 deadline: 1679399939199, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:59,267 DEBUG [Thread-2783] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:59,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:59,268 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:57:59,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:57:59,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:57:59,269 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:57:59,269 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:59,269 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:57:59,269 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:57:59,270 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:59,270 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:59,270 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:59,270 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:59,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:59,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:59,270 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:59,271 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:57:59,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:59,271 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:59,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:57:59,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:59,271 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:57:59,271 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:57:59,271 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:57:59,271 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:59,271 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:57:59,271 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:57:59,272 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:57:59,272 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:59,272 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. started... 2023-03-21 11:57:59,272 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. as already flushing 2023-03-21 11:57:59,272 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:59,272 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:57:59,272 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:57:59,272 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:59,272 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:57:59,272 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:57:59,273 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:59,273 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:59,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:59,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:59,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:59,273 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:59,273 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:57:59,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:59,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:59,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:59,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:59,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:59,274 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:57:59,274 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d292160[Count = 0] remaining members to acquire global barrier 2023-03-21 11:57:59,274 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:57:59,274 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:59,274 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:59,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:59,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:59,275 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:57:59,275 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:57:59,275 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:57:59,275 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:59,275 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:57:59,275 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:57:59,275 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:59,275 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:59,275 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:57:59,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:59,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:59,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:59,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:59,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:59,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:59,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:59,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:59,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:59,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:59,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:57:59,277 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:57:59,277 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:57:59,277 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:57:59,277 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:57:59,277 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:57:59,277 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:57:59,278 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:59,278 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:59,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:59,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:59,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:59,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:59,278 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:59,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:59,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:59,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:59,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:59,278 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:59,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:59,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:59,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:59,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:59,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:59,279 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:59,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:59,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:59,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:59,281 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:59,281 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:59,281 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:59,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:59,281 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:59,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:59,281 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:59,281 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:59,281 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:59,281 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:57:59,281 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:57:59,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:59,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:59,281 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:59,281 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:59,281 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:59,281 DEBUG [Thread-2783] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:57:59,281 DEBUG [Thread-2783] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:57:59,330 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=366 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/B/0489fe4d13aa4add99feea3d3a580809 2023-03-21 11:57:59,381 DEBUG [Thread-2783] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:59,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:59,383 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:57:59,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:57:59,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:57:59,384 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:57:59,384 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:59,384 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:57:59,384 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:57:59,384 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:59,384 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:59,385 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:59,385 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:59,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:59,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:59,385 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:59,385 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:57:59,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:59,385 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:59,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:57:59,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:59,386 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:57:59,386 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:57:59,386 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:57:59,386 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:57:59,386 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:59,386 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:57:59,394 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:57:59,394 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:59,394 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. started... 2023-03-21 11:57:59,394 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:57:59,394 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. as already flushing 2023-03-21 11:57:59,394 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:59,394 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:57:59,394 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:57:59,394 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:59,394 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:57:59,394 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:57:59,403 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:59,403 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:59,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:59,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:59,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:59,403 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:59,403 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:57:59,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:59,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:59,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:59,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:59,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:59,404 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:57:59,404 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a15e704[Count = 0] remaining members to acquire global barrier 2023-03-21 11:57:59,404 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:57:59,404 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:59,407 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:59,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:59,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:59,407 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:59,407 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:57:59,407 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:57:59,407 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:57:59,407 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:57:59,408 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:59,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:59,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:59,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:59,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:59,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:59,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:59,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:59,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:59,409 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:57:59,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:59,409 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:59,409 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:57:59,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:59,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:57:59,410 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:57:59,410 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:57:59,410 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:57:59,410 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:57:59,410 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:57:59,410 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:57:59,410 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:59,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:59,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:59,411 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:59,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:59,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:59,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:59,411 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:59,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:59,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:59,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:59,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:59,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:59,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:59,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:59,412 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:59,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:59,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:59,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:59,413 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:59,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:59,414 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:59,414 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:59,414 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:59,414 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:59,414 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:57:59,414 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:57:59,414 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:59,414 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:59,414 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:59,414 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:59,414 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:59,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:59,414 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:59,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:59,414 DEBUG [Thread-2783] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:57:59,414 DEBUG [Thread-2783] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:57:59,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:59,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:59,514 DEBUG [Thread-2783] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:59,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:59,516 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:57:59,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:57:59,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:57:59,517 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:57:59,517 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:59,517 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:57:59,517 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:57:59,517 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:59,517 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:59,518 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:59,518 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:59,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:59,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:59,518 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:59,518 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:57:59,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:59,518 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:59,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:57:59,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:59,519 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:57:59,519 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:57:59,519 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:57:59,519 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:57:59,519 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:57:59,519 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:57:59,519 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:57:59,519 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:57:59,519 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:59,519 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. started... 2023-03-21 11:57:59,520 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. as already flushing 2023-03-21 11:57:59,520 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:59,520 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:57:59,520 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:57:59,520 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:59,520 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:57:59,520 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:57:59,520 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:59,520 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:59,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:59,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:59,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:59,521 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:59,521 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:57:59,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:59,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:59,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:59,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:59,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:59,522 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:57:59,522 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@71817c82[Count = 0] remaining members to acquire global barrier 2023-03-21 11:57:59,522 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:57:59,522 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:59,522 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:59,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:59,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:59,522 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:57:59,522 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:57:59,522 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:59,522 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:57:59,522 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:57:59,523 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:59,523 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:57:59,523 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:59,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:59,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:59,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:59,523 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:57:59,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:59,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:59,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:59,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:59,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:59,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:59,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:59,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:57:59,526 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:57:59,526 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:57:59,526 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:57:59,526 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:57:59,526 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:57:59,526 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:57:59,527 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:59,527 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:59,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:59,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:59,527 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:59,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:59,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:57:59,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:57:59,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:59,527 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:59,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:59,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:59,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:57:59,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:59,528 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:59,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:57:59,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:59,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:59,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:57:59,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:57:59,530 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:57:59,530 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:59,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:57:59,530 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:59,530 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:57:59,530 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:57:59,530 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:57:59,530 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:57:59,530 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:57:59,530 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:59,530 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:59,530 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:57:59,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:57:59,530 DEBUG [Thread-2783] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:57:59,530 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:57:59,530 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:57:59,530 DEBUG [Thread-2783] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:57:59,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:57:59,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:57:59,628 DEBUG [Thread-2784] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x62447900 to 127.0.0.1:61761 2023-03-21 11:57:59,628 DEBUG [Thread-2784] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-21 11:57:59,630 DEBUG [Thread-2783] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:57:59,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:57:59,632 DEBUG [Thread-2786] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x519db5f2 to 127.0.0.1:61761 2023-03-21 11:57:59,632 DEBUG [Thread-2786] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-21 11:57:59,633 DEBUG [Thread-2790] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x22bf9ddd to 127.0.0.1:61761 2023-03-21 11:57:59,633 DEBUG [Thread-2790] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-21 11:57:59,635 DEBUG [Thread-2788] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5c9c1feb to 127.0.0.1:61761 2023-03-21 11:57:59,635 DEBUG [Thread-2788] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-21 11:57:59,638 DEBUG [Thread-2792] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x34c4ae9a to 127.0.0.1:61761 2023-03-21 11:57:59,638 DEBUG [Thread-2792] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-21 11:57:59,704 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:59,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48302 deadline: 1679399939704, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:59,705 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:59,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 276 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48362 deadline: 1679399939705, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:59,707 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:57:59,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48284 deadline: 1679399939706, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:57:59,739 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=366 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/C/3e6bd035f78748d780f56036691e83d2 2023-03-21 11:57:59,749 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/A/a4083b568dad40819a4740c449a3d8e1 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/a4083b568dad40819a4740c449a3d8e1 2023-03-21 11:57:59,753 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/a4083b568dad40819a4740c449a3d8e1, entries=200, sequenceid=366, filesize=14.3 K 2023-03-21 11:57:59,754 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/B/0489fe4d13aa4add99feea3d3a580809 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/0489fe4d13aa4add99feea3d3a580809 2023-03-21 11:57:59,758 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/0489fe4d13aa4add99feea3d3a580809, entries=150, sequenceid=366, filesize=11.9 K 2023-03-21 11:57:59,759 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/C/3e6bd035f78748d780f56036691e83d2 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/3e6bd035f78748d780f56036691e83d2 2023-03-21 11:57:59,764 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/3e6bd035f78748d780f56036691e83d2, entries=150, sequenceid=366, filesize=11.9 K 2023-03-21 11:57:59,764 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=140.89 KB/144270 for 8dbd4df2f9769e1bcf08d64a8aceb9ad in 1248ms, sequenceid=366, compaction requested=true 2023-03-21 11:57:59,764 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8dbd4df2f9769e1bcf08d64a8aceb9ad: 2023-03-21 11:57:59,765 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-21 11:57:59,765 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-21 11:57:59,765 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-21 11:57:59,765 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-21 11:57:59,765 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-21 11:57:59,766 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39782 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-21 11:57:59,766 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1912): 8dbd4df2f9769e1bcf08d64a8aceb9ad/A is initiating minor compaction (all files) 2023-03-21 11:57:59,766 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37342 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-21 11:57:59,766 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8dbd4df2f9769e1bcf08d64a8aceb9ad/A in TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:59,766 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1912): 8dbd4df2f9769e1bcf08d64a8aceb9ad/B is initiating minor compaction (all files) 2023-03-21 11:57:59,766 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/99b529bc783042bbbad5234c5bd1ec62, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/5d821ecd508a4491b4a9f7cfe200d067, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/a4083b568dad40819a4740c449a3d8e1] into tmpdir=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp, totalSize=38.8 K 2023-03-21 11:57:59,766 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8dbd4df2f9769e1bcf08d64a8aceb9ad/B in TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:57:59,767 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/6a87bfe9de63438cba29724c00e3692b, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/8d8f54ca750f481798cc77d788b87ae0, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/0489fe4d13aa4add99feea3d3a580809] into tmpdir=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp, totalSize=36.5 K 2023-03-21 11:57:59,767 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 99b529bc783042bbbad5234c5bd1ec62, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=334, earliestPutTs=1679399875727 2023-03-21 11:57:59,767 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 6a87bfe9de63438cba29724c00e3692b, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=334, earliestPutTs=1679399875727 2023-03-21 11:57:59,767 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 5d821ecd508a4491b4a9f7cfe200d067, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=350, earliestPutTs=1679399876363 2023-03-21 11:57:59,767 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 8d8f54ca750f481798cc77d788b87ae0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=350, earliestPutTs=1679399876363 2023-03-21 11:57:59,768 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting a4083b568dad40819a4740c449a3d8e1, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=366, earliestPutTs=1679399878510 2023-03-21 11:57:59,768 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 0489fe4d13aa4add99feea3d3a580809, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=366, earliestPutTs=1679399878510 2023-03-21 11:57:59,775 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8dbd4df2f9769e1bcf08d64a8aceb9ad#B#compaction#424 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 57.69 MB/second 2023-03-21 11:57:59,775 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] throttle.PressureAwareThroughputController(145): 8dbd4df2f9769e1bcf08d64a8aceb9ad#A#compaction#423 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 57.69 MB/second 2023-03-21 11:58:00,189 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/B/6c4d25d70e584290b7b43af08b98c730 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/6c4d25d70e584290b7b43af08b98c730 2023-03-21 11:58:00,190 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/A/002b492a7a064e5e902276e253c70b27 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/002b492a7a064e5e902276e253c70b27 2023-03-21 11:58:00,196 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8dbd4df2f9769e1bcf08d64a8aceb9ad/A of 8dbd4df2f9769e1bcf08d64a8aceb9ad into 002b492a7a064e5e902276e253c70b27(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-21 11:58:00,196 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8dbd4df2f9769e1bcf08d64a8aceb9ad/B of 8dbd4df2f9769e1bcf08d64a8aceb9ad into 6c4d25d70e584290b7b43af08b98c730(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-21 11:58:00,196 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8dbd4df2f9769e1bcf08d64a8aceb9ad: 2023-03-21 11:58:00,196 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8dbd4df2f9769e1bcf08d64a8aceb9ad: 2023-03-21 11:58:00,196 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad., storeName=8dbd4df2f9769e1bcf08d64a8aceb9ad/A, priority=13, startTime=1679399879765; duration=0sec 2023-03-21 11:58:00,196 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad., storeName=8dbd4df2f9769e1bcf08d64a8aceb9ad/B, priority=13, startTime=1679399879765; duration=0sec 2023-03-21 11:58:00,196 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-21 11:58:00,196 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-21 11:58:00,196 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-21 11:58:00,198 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37342 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-21 11:58:00,198 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1912): 8dbd4df2f9769e1bcf08d64a8aceb9ad/C is initiating minor compaction (all files) 2023-03-21 11:58:00,198 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8dbd4df2f9769e1bcf08d64a8aceb9ad/C in TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:58:00,198 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/381d152397b3409699f3688d0fc8048f, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/016f5e93911c42df83d3f00d4fad3c86, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/3e6bd035f78748d780f56036691e83d2] into tmpdir=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp, totalSize=36.5 K 2023-03-21 11:58:00,198 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 381d152397b3409699f3688d0fc8048f, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=334, earliestPutTs=1679399875727 2023-03-21 11:58:00,198 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 016f5e93911c42df83d3f00d4fad3c86, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=350, earliestPutTs=1679399876363 2023-03-21 11:58:00,199 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 3e6bd035f78748d780f56036691e83d2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=366, earliestPutTs=1679399878510 2023-03-21 11:58:00,203 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] throttle.PressureAwareThroughputController(145): 8dbd4df2f9769e1bcf08d64a8aceb9ad#C#compaction#425 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 57.69 MB/second 2023-03-21 11:58:00,617 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/C/d358874b64c945839a9ae78ba282bd26 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/d358874b64c945839a9ae78ba282bd26 2023-03-21 11:58:00,622 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8dbd4df2f9769e1bcf08d64a8aceb9ad/C of 8dbd4df2f9769e1bcf08d64a8aceb9ad into d358874b64c945839a9ae78ba282bd26(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-21 11:58:00,622 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8dbd4df2f9769e1bcf08d64a8aceb9ad: 2023-03-21 11:58:00,622 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad., storeName=8dbd4df2f9769e1bcf08d64a8aceb9ad/C, priority=13, startTime=1679399879765; duration=0sec 2023-03-21 11:58:00,622 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-21 11:58:00,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(9158): Flush requested on 8dbd4df2f9769e1bcf08d64a8aceb9ad 2023-03-21 11:58:00,715 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8dbd4df2f9769e1bcf08d64a8aceb9ad 3/3 column families, dataSize=147.60 KB heapSize=415.45 KB 2023-03-21 11:58:00,716 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dbd4df2f9769e1bcf08d64a8aceb9ad, store=A 2023-03-21 11:58:00,716 DEBUG [Thread-2777] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x50e89dd2 to 127.0.0.1:61761 2023-03-21 11:58:00,716 DEBUG [Thread-2777] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-21 11:58:00,716 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-21 11:58:00,717 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dbd4df2f9769e1bcf08d64a8aceb9ad, store=B 2023-03-21 11:58:00,717 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-21 11:58:00,717 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dbd4df2f9769e1bcf08d64a8aceb9ad, store=C 2023-03-21 11:58:00,717 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-21 11:58:00,720 DEBUG [Thread-2775] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7cb49278 to 127.0.0.1:61761 2023-03-21 11:58:00,720 DEBUG [Thread-2775] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-21 11:58:00,720 DEBUG [Thread-2773] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3e21fa39 to 127.0.0.1:61761 2023-03-21 11:58:00,720 DEBUG [Thread-2773] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-21 11:58:00,856 DEBUG [Thread-2781] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4c4d3f2e to 127.0.0.1:61761 2023-03-21 11:58:00,856 DEBUG [Thread-2781] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-21 11:58:00,894 DEBUG [Thread-2779] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x545adde0 to 127.0.0.1:61761 2023-03-21 11:58:00,894 DEBUG [Thread-2779] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-21 11:58:00,894 INFO [Listener at localhost.localdomain/37475] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-03-21 11:58:00,895 INFO [Listener at localhost.localdomain/37475] hbase.AcidGuaranteesTestTool(394): wrote 70 2023-03-21 11:58:00,895 INFO [Listener at localhost.localdomain/37475] hbase.AcidGuaranteesTestTool(394): wrote 69 2023-03-21 11:58:00,895 INFO [Listener at localhost.localdomain/37475] hbase.AcidGuaranteesTestTool(394): wrote 153 2023-03-21 11:58:00,895 INFO [Listener at localhost.localdomain/37475] hbase.AcidGuaranteesTestTool(394): wrote 16 2023-03-21 11:58:00,895 INFO [Listener at localhost.localdomain/37475] hbase.AcidGuaranteesTestTool(394): wrote 14 2023-03-21 11:58:00,895 INFO [Listener at localhost.localdomain/37475] hbase.AcidGuaranteesTestTool(396): Readers: 2023-03-21 11:58:00,896 INFO [Listener at localhost.localdomain/37475] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-03-21 11:58:00,896 INFO [Listener at localhost.localdomain/37475] hbase.AcidGuaranteesTestTool(402): scanned 2230 2023-03-21 11:58:00,896 INFO [Listener at localhost.localdomain/37475] hbase.AcidGuaranteesTestTool(403): verified 6687 rows 2023-03-21 11:58:00,896 INFO [Listener at localhost.localdomain/37475] hbase.AcidGuaranteesTestTool(402): scanned 2220 2023-03-21 11:58:00,896 INFO [Listener at localhost.localdomain/37475] hbase.AcidGuaranteesTestTool(403): verified 6660 rows 2023-03-21 11:58:00,896 INFO [Listener at localhost.localdomain/37475] hbase.AcidGuaranteesTestTool(402): scanned 2232 2023-03-21 11:58:00,896 INFO [Listener at localhost.localdomain/37475] hbase.AcidGuaranteesTestTool(403): verified 6696 rows 2023-03-21 11:58:00,896 INFO [Listener at localhost.localdomain/37475] hbase.AcidGuaranteesTestTool(402): scanned 2222 2023-03-21 11:58:00,897 INFO [Listener at localhost.localdomain/37475] hbase.AcidGuaranteesTestTool(403): verified 6666 rows 2023-03-21 11:58:00,897 INFO [Listener at localhost.localdomain/37475] hbase.AcidGuaranteesTestTool(402): scanned 2225 2023-03-21 11:58:00,897 INFO [Listener at localhost.localdomain/37475] hbase.AcidGuaranteesTestTool(403): verified 6675 rows 2023-03-21 11:58:00,897 INFO [Listener at localhost.localdomain/37475] client.ConnectionImplementation(1959): Closing master protocol: MasterService 2023-03-21 11:58:00,897 DEBUG [Listener at localhost.localdomain/37475] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x17a38ae8 to 127.0.0.1:61761 2023-03-21 11:58:00,897 DEBUG [Listener at localhost.localdomain/37475] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-21 11:58:00,903 INFO [Listener at localhost.localdomain/37475] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-03-21 11:58:00,903 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.HMaster$11(2418): Client=jenkins//148.251.75.209 disable TestAcidGuarantees 2023-03-21 11:58:00,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure2.ProcedureExecutor(1029): Stored pid=50, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-03-21 11:58:00,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-03-21 11:58:00,906 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679399880906"}]},"ts":"1679399880906"} 2023-03-21 11:58:00,907 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-03-21 11:58:00,911 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-03-21 11:58:00,912 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=51, ppid=50, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=8dbd4df2f9769e1bcf08d64a8aceb9ad, UNASSIGN}] 2023-03-21 11:58:00,912 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=51, ppid=50, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=8dbd4df2f9769e1bcf08d64a8aceb9ad, UNASSIGN 2023-03-21 11:58:00,913 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=8dbd4df2f9769e1bcf08d64a8aceb9ad, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:00,913 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679399880913"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1679399880913"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679399880913"}]},"ts":"1679399880913"} 2023-03-21 11:58:00,913 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=52, ppid=51, state=RUNNABLE; CloseRegionProcedure 8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030}] 2023-03-21 11:58:01,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-03-21 11:58:01,066 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 8dbd4df2f9769e1bcf08d64a8aceb9ad 2023-03-21 11:58:01,066 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 8dbd4df2f9769e1bcf08d64a8aceb9ad, disabling compactions & flushes 2023-03-21 11:58:01,066 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1858): waiting for 0 compactions & cache flush to complete for region TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:58:01,126 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=406 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/A/2f68802f9a96460182d9b2f1085e94ec 2023-03-21 11:58:01,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-03-21 11:58:01,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-03-21 11:58:01,545 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=406 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/B/706a998f531b49d795e6c6c8918e28ee 2023-03-21 11:58:01,964 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=406 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/C/85482d180e0443d6beab1e99126a0295 2023-03-21 11:58:01,969 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/A/2f68802f9a96460182d9b2f1085e94ec as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/2f68802f9a96460182d9b2f1085e94ec 2023-03-21 11:58:01,974 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/2f68802f9a96460182d9b2f1085e94ec, entries=150, sequenceid=406, filesize=11.9 K 2023-03-21 11:58:01,975 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/B/706a998f531b49d795e6c6c8918e28ee as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/706a998f531b49d795e6c6c8918e28ee 2023-03-21 11:58:01,980 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/706a998f531b49d795e6c6c8918e28ee, entries=150, sequenceid=406, filesize=11.9 K 2023-03-21 11:58:01,981 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/C/85482d180e0443d6beab1e99126a0295 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/85482d180e0443d6beab1e99126a0295 2023-03-21 11:58:01,986 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/85482d180e0443d6beab1e99126a0295, entries=150, sequenceid=406, filesize=11.9 K 2023-03-21 11:58:01,987 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~415.41 KB/425376, currentSize=26.84 KB/27480 for 8dbd4df2f9769e1bcf08d64a8aceb9ad in 1272ms, sequenceid=406, compaction requested=false 2023-03-21 11:58:01,987 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8dbd4df2f9769e1bcf08d64a8aceb9ad: 2023-03-21 11:58:01,987 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:58:01,987 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:58:01,987 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. after waiting 0 ms 2023-03-21 11:58:01,987 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:58:01,987 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing 8dbd4df2f9769e1bcf08d64a8aceb9ad 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-03-21 11:58:01,988 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dbd4df2f9769e1bcf08d64a8aceb9ad, store=A 2023-03-21 11:58:01,988 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:58:01,988 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dbd4df2f9769e1bcf08d64a8aceb9ad, store=B 2023-03-21 11:58:01,988 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:58:01,988 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8dbd4df2f9769e1bcf08d64a8aceb9ad, store=C 2023-03-21 11:58:01,988 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:58:02,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-03-21 11:58:02,396 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=413 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/A/8c6848e74fe745ddbac2d979802821eb 2023-03-21 11:58:02,811 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=413 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/B/5f7075dd214b45cab2ec623699179f49 2023-03-21 11:58:03,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-03-21 11:58:03,228 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=413 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/C/169ee5ef6ecf4cebb4608e796d953e82 2023-03-21 11:58:03,238 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/A/8c6848e74fe745ddbac2d979802821eb as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/8c6848e74fe745ddbac2d979802821eb 2023-03-21 11:58:03,243 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/8c6848e74fe745ddbac2d979802821eb, entries=100, sequenceid=413, filesize=9.5 K 2023-03-21 11:58:03,244 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/B/5f7075dd214b45cab2ec623699179f49 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/5f7075dd214b45cab2ec623699179f49 2023-03-21 11:58:03,251 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/5f7075dd214b45cab2ec623699179f49, entries=100, sequenceid=413, filesize=9.5 K 2023-03-21 11:58:03,252 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/.tmp/C/169ee5ef6ecf4cebb4608e796d953e82 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/169ee5ef6ecf4cebb4608e796d953e82 2023-03-21 11:58:03,258 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/169ee5ef6ecf4cebb4608e796d953e82, entries=100, sequenceid=413, filesize=9.5 K 2023-03-21 11:58:03,259 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for 8dbd4df2f9769e1bcf08d64a8aceb9ad in 1272ms, sequenceid=413, compaction requested=true 2023-03-21 11:58:03,261 DEBUG [StoreCloser-TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/48f70e5740e24bf2b2ce7bf8b98eea79, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/7a7ab837338a4b68bac517fbb992affa, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/46401f06a61b4f9c9189fc6dacf34687, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/0e54f1c8c20443339da361430aebe1fa, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/bb0436b397b24ef58e44d776c40f73bc, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/d18ce65c0770425b9f9bf47b5f20fe8a, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/118e5d59e8a04a8bb6610dae3e71dc70, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/93fb6dc50f6543aab2b0b2cb1426906b, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/67d31c5bbffc4ef98bec224823310122, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/906ec9a526024ace9409a51f7d3dbc5c, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/dceaf26e29fb4b62961d135a5f226aed, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/2abf421417db453f89a497047c79f2f2, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/1181414af9ad49f19714eb5b930ab75b, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/36329bbf325443dfa9e3a8b822518aa3, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/2a34d28534194ec2ac7a4a62a0030e51, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/43d97de7b1554db4861deaa29dc22a0d, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/3968a80e65f849fc8ad24bf53fb688a4, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/a0ce75904f18473988823ac43e192f8f, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/71ea227e90e74850a2392ce48309483a, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/ebd74e81e764452f8fa946cf71020c40, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/cfa8be38c48a42ab8840f75e842687d2, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/04ab5fe013db4e7db0f76dca58215114, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/99b529bc783042bbbad5234c5bd1ec62, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/512dd7bccfc94ab58c1730d5deb22642, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/5d821ecd508a4491b4a9f7cfe200d067, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/a4083b568dad40819a4740c449a3d8e1] to archive 2023-03-21 11:58:03,262 DEBUG [StoreCloser-TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-21 11:58:03,264 DEBUG [StoreCloser-TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/48f70e5740e24bf2b2ce7bf8b98eea79 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/48f70e5740e24bf2b2ce7bf8b98eea79 2023-03-21 11:58:03,266 DEBUG [StoreCloser-TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/7a7ab837338a4b68bac517fbb992affa to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/7a7ab837338a4b68bac517fbb992affa 2023-03-21 11:58:03,268 DEBUG [StoreCloser-TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/46401f06a61b4f9c9189fc6dacf34687 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/46401f06a61b4f9c9189fc6dacf34687 2023-03-21 11:58:03,270 DEBUG [StoreCloser-TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/0e54f1c8c20443339da361430aebe1fa to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/0e54f1c8c20443339da361430aebe1fa 2023-03-21 11:58:03,271 DEBUG [StoreCloser-TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/bb0436b397b24ef58e44d776c40f73bc to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/bb0436b397b24ef58e44d776c40f73bc 2023-03-21 11:58:03,272 DEBUG [StoreCloser-TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/d18ce65c0770425b9f9bf47b5f20fe8a to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/d18ce65c0770425b9f9bf47b5f20fe8a 2023-03-21 11:58:03,273 DEBUG [StoreCloser-TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/118e5d59e8a04a8bb6610dae3e71dc70 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/118e5d59e8a04a8bb6610dae3e71dc70 2023-03-21 11:58:03,274 DEBUG [StoreCloser-TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/93fb6dc50f6543aab2b0b2cb1426906b to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/93fb6dc50f6543aab2b0b2cb1426906b 2023-03-21 11:58:03,276 DEBUG [StoreCloser-TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/67d31c5bbffc4ef98bec224823310122 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/67d31c5bbffc4ef98bec224823310122 2023-03-21 11:58:03,277 DEBUG [StoreCloser-TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/906ec9a526024ace9409a51f7d3dbc5c to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/906ec9a526024ace9409a51f7d3dbc5c 2023-03-21 11:58:03,278 DEBUG [StoreCloser-TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/dceaf26e29fb4b62961d135a5f226aed to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/dceaf26e29fb4b62961d135a5f226aed 2023-03-21 11:58:03,279 DEBUG [StoreCloser-TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/2abf421417db453f89a497047c79f2f2 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/2abf421417db453f89a497047c79f2f2 2023-03-21 11:58:03,280 DEBUG [StoreCloser-TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/1181414af9ad49f19714eb5b930ab75b to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/1181414af9ad49f19714eb5b930ab75b 2023-03-21 11:58:03,281 DEBUG [StoreCloser-TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/36329bbf325443dfa9e3a8b822518aa3 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/36329bbf325443dfa9e3a8b822518aa3 2023-03-21 11:58:03,282 DEBUG [StoreCloser-TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/2a34d28534194ec2ac7a4a62a0030e51 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/2a34d28534194ec2ac7a4a62a0030e51 2023-03-21 11:58:03,283 DEBUG [StoreCloser-TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/43d97de7b1554db4861deaa29dc22a0d to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/43d97de7b1554db4861deaa29dc22a0d 2023-03-21 11:58:03,284 DEBUG [StoreCloser-TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/3968a80e65f849fc8ad24bf53fb688a4 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/3968a80e65f849fc8ad24bf53fb688a4 2023-03-21 11:58:03,285 DEBUG [StoreCloser-TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/a0ce75904f18473988823ac43e192f8f to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/a0ce75904f18473988823ac43e192f8f 2023-03-21 11:58:03,286 DEBUG [StoreCloser-TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/71ea227e90e74850a2392ce48309483a to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/71ea227e90e74850a2392ce48309483a 2023-03-21 11:58:03,287 DEBUG [StoreCloser-TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/ebd74e81e764452f8fa946cf71020c40 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/ebd74e81e764452f8fa946cf71020c40 2023-03-21 11:58:03,288 DEBUG [StoreCloser-TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/cfa8be38c48a42ab8840f75e842687d2 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/cfa8be38c48a42ab8840f75e842687d2 2023-03-21 11:58:03,289 DEBUG [StoreCloser-TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/04ab5fe013db4e7db0f76dca58215114 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/04ab5fe013db4e7db0f76dca58215114 2023-03-21 11:58:03,290 DEBUG [StoreCloser-TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/99b529bc783042bbbad5234c5bd1ec62 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/99b529bc783042bbbad5234c5bd1ec62 2023-03-21 11:58:03,291 DEBUG [StoreCloser-TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/512dd7bccfc94ab58c1730d5deb22642 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/512dd7bccfc94ab58c1730d5deb22642 2023-03-21 11:58:03,292 DEBUG [StoreCloser-TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/5d821ecd508a4491b4a9f7cfe200d067 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/5d821ecd508a4491b4a9f7cfe200d067 2023-03-21 11:58:03,293 DEBUG [StoreCloser-TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/a4083b568dad40819a4740c449a3d8e1 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/a4083b568dad40819a4740c449a3d8e1 2023-03-21 11:58:03,296 DEBUG [StoreCloser-TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/efdec08f12ce4e268b133e793464c267, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/52168d3b40b74cde9f216a960c66fab0, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/1848cf8a974748bcbfe3f9f7e952b709, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/ee565f7fbdee45298a18bef5464d4206, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/dc37a1f2797d499ea1d6e8aa8a1367e4, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/a53e2fa6af314745818d764952284452, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/0e62ed6012624fa889f1cbd819384503, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/da5e172d03fd4ff2822262d9d22d3ea4, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/349ea59266304913a63e4d1e0a769cd6, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/3913b29f536045c09825e00e42f26792, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/3a398d0612d04e3bb07f16f19e8a52cf, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/f098f9d7c2a64c2c89b776f2be1b4c5d, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/c307cf7252954120a77104dd171f9519, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/6ad2d455bdc245f9ab2049fc0d4d50ee, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/d0f86ae6ccc94bf39fdd3c3b2a8d29c6, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/867fe675231f425a94dfce964a19a4b2, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/8d345ea0ff1c4b819d11944cd5014728, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/400c2550f06f49cb90379e0342216521, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/1a3eddc851604db9b8524ea5b849a2f6, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/aedadcc0d1ea4749bddf13d90362a9b5, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/31934ad7b1e5408fa25560c0cc70ae67, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/e4a5d2e9c3644955a3f9e0106d147a4c, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/6a87bfe9de63438cba29724c00e3692b, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/2da5692354da4dbba2be59bdaa1410a7, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/8d8f54ca750f481798cc77d788b87ae0, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/0489fe4d13aa4add99feea3d3a580809] to archive 2023-03-21 11:58:03,296 DEBUG [StoreCloser-TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-21 11:58:03,298 DEBUG [StoreCloser-TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/efdec08f12ce4e268b133e793464c267 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/efdec08f12ce4e268b133e793464c267 2023-03-21 11:58:03,299 DEBUG [StoreCloser-TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/52168d3b40b74cde9f216a960c66fab0 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/52168d3b40b74cde9f216a960c66fab0 2023-03-21 11:58:03,301 DEBUG [StoreCloser-TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/1848cf8a974748bcbfe3f9f7e952b709 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/1848cf8a974748bcbfe3f9f7e952b709 2023-03-21 11:58:03,303 DEBUG [StoreCloser-TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/ee565f7fbdee45298a18bef5464d4206 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/ee565f7fbdee45298a18bef5464d4206 2023-03-21 11:58:03,304 DEBUG [StoreCloser-TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/dc37a1f2797d499ea1d6e8aa8a1367e4 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/dc37a1f2797d499ea1d6e8aa8a1367e4 2023-03-21 11:58:03,306 DEBUG [StoreCloser-TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/a53e2fa6af314745818d764952284452 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/a53e2fa6af314745818d764952284452 2023-03-21 11:58:03,307 DEBUG [StoreCloser-TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/0e62ed6012624fa889f1cbd819384503 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/0e62ed6012624fa889f1cbd819384503 2023-03-21 11:58:03,309 DEBUG [StoreCloser-TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/da5e172d03fd4ff2822262d9d22d3ea4 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/da5e172d03fd4ff2822262d9d22d3ea4 2023-03-21 11:58:03,311 DEBUG [StoreCloser-TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/349ea59266304913a63e4d1e0a769cd6 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/349ea59266304913a63e4d1e0a769cd6 2023-03-21 11:58:03,312 DEBUG [StoreCloser-TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/3913b29f536045c09825e00e42f26792 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/3913b29f536045c09825e00e42f26792 2023-03-21 11:58:03,314 DEBUG [StoreCloser-TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/3a398d0612d04e3bb07f16f19e8a52cf to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/3a398d0612d04e3bb07f16f19e8a52cf 2023-03-21 11:58:03,315 DEBUG [StoreCloser-TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/f098f9d7c2a64c2c89b776f2be1b4c5d to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/f098f9d7c2a64c2c89b776f2be1b4c5d 2023-03-21 11:58:03,317 DEBUG [StoreCloser-TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/c307cf7252954120a77104dd171f9519 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/c307cf7252954120a77104dd171f9519 2023-03-21 11:58:03,319 DEBUG [StoreCloser-TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/6ad2d455bdc245f9ab2049fc0d4d50ee to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/6ad2d455bdc245f9ab2049fc0d4d50ee 2023-03-21 11:58:03,320 DEBUG [StoreCloser-TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/d0f86ae6ccc94bf39fdd3c3b2a8d29c6 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/d0f86ae6ccc94bf39fdd3c3b2a8d29c6 2023-03-21 11:58:03,322 DEBUG [StoreCloser-TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/867fe675231f425a94dfce964a19a4b2 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/867fe675231f425a94dfce964a19a4b2 2023-03-21 11:58:03,323 DEBUG [StoreCloser-TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/8d345ea0ff1c4b819d11944cd5014728 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/8d345ea0ff1c4b819d11944cd5014728 2023-03-21 11:58:03,325 DEBUG [StoreCloser-TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/400c2550f06f49cb90379e0342216521 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/400c2550f06f49cb90379e0342216521 2023-03-21 11:58:03,327 DEBUG [StoreCloser-TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/1a3eddc851604db9b8524ea5b849a2f6 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/1a3eddc851604db9b8524ea5b849a2f6 2023-03-21 11:58:03,328 DEBUG [StoreCloser-TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/aedadcc0d1ea4749bddf13d90362a9b5 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/aedadcc0d1ea4749bddf13d90362a9b5 2023-03-21 11:58:03,330 DEBUG [StoreCloser-TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/31934ad7b1e5408fa25560c0cc70ae67 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/31934ad7b1e5408fa25560c0cc70ae67 2023-03-21 11:58:03,332 DEBUG [StoreCloser-TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/e4a5d2e9c3644955a3f9e0106d147a4c to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/e4a5d2e9c3644955a3f9e0106d147a4c 2023-03-21 11:58:03,334 DEBUG [StoreCloser-TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/6a87bfe9de63438cba29724c00e3692b to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/6a87bfe9de63438cba29724c00e3692b 2023-03-21 11:58:03,335 DEBUG [StoreCloser-TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/2da5692354da4dbba2be59bdaa1410a7 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/2da5692354da4dbba2be59bdaa1410a7 2023-03-21 11:58:03,337 DEBUG [StoreCloser-TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/8d8f54ca750f481798cc77d788b87ae0 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/8d8f54ca750f481798cc77d788b87ae0 2023-03-21 11:58:03,339 DEBUG [StoreCloser-TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/0489fe4d13aa4add99feea3d3a580809 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/0489fe4d13aa4add99feea3d3a580809 2023-03-21 11:58:03,343 DEBUG [StoreCloser-TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/3ad32817e68d4feda55ff068caa3304d, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/ba5820a8f0f045988e1819b794743b47, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/c2558cdf5d6b4f219c5d7890dc754e3f, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/c59afd9df3de4dad85f72d1caa39551f, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/f43be7b7323c479c98d2ce6f2254d49d, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/06cdbe75ba0642bcac8ccdaa2114def6, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/f16c7a152162417982b660dc6ae0f1c5, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/b962320e8ae24525bed2262267651f53, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/b51026cea1e14b69a357d1f2a48cfbd0, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/86044c66c16e4ac78a4fad9ec58c6d99, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/b35cf28e8e7146b6899cfa8fc5fe48ec, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/6401b28e4e6a459baeed257a89acb86b, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/059653d6da58499db49c19ab54892bad, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/0d4f9c1fb71f44ab8d48af6ea030f1da, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/966966a35deb43208cb7cb1f704fdd06, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/7245a8d75dab4ef59f461ab0deab300f, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/8e3e6a1fac1d44eaa56ce057b6fbffc9, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/cc95426bd4c94e92bf5a42ddd60409b8, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/54769945dc7f459d9c2ad0de3690f9e4, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/4d5963850eb840a0996bfe057f5ca4f2, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/29dbd806d71c4ad8879863662b40cf91, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/6ff5db78cc884b948cf4c91a58b5c3bb, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/381d152397b3409699f3688d0fc8048f, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/3aeff156e4274296bae04c4b64f94ac8, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/016f5e93911c42df83d3f00d4fad3c86, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/3e6bd035f78748d780f56036691e83d2] to archive 2023-03-21 11:58:03,343 DEBUG [StoreCloser-TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-21 11:58:03,345 DEBUG [StoreCloser-TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/3ad32817e68d4feda55ff068caa3304d to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/3ad32817e68d4feda55ff068caa3304d 2023-03-21 11:58:03,346 DEBUG [StoreCloser-TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/ba5820a8f0f045988e1819b794743b47 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/ba5820a8f0f045988e1819b794743b47 2023-03-21 11:58:03,348 DEBUG [StoreCloser-TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/c2558cdf5d6b4f219c5d7890dc754e3f to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/c2558cdf5d6b4f219c5d7890dc754e3f 2023-03-21 11:58:03,349 DEBUG [StoreCloser-TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/c59afd9df3de4dad85f72d1caa39551f to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/c59afd9df3de4dad85f72d1caa39551f 2023-03-21 11:58:03,350 DEBUG [StoreCloser-TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/f43be7b7323c479c98d2ce6f2254d49d to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/f43be7b7323c479c98d2ce6f2254d49d 2023-03-21 11:58:03,351 DEBUG [StoreCloser-TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/06cdbe75ba0642bcac8ccdaa2114def6 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/06cdbe75ba0642bcac8ccdaa2114def6 2023-03-21 11:58:03,353 DEBUG [StoreCloser-TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/f16c7a152162417982b660dc6ae0f1c5 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/f16c7a152162417982b660dc6ae0f1c5 2023-03-21 11:58:03,354 DEBUG [StoreCloser-TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/b962320e8ae24525bed2262267651f53 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/b962320e8ae24525bed2262267651f53 2023-03-21 11:58:03,355 DEBUG [StoreCloser-TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/b51026cea1e14b69a357d1f2a48cfbd0 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/b51026cea1e14b69a357d1f2a48cfbd0 2023-03-21 11:58:03,357 DEBUG [StoreCloser-TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/86044c66c16e4ac78a4fad9ec58c6d99 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/86044c66c16e4ac78a4fad9ec58c6d99 2023-03-21 11:58:03,358 DEBUG [StoreCloser-TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/b35cf28e8e7146b6899cfa8fc5fe48ec to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/b35cf28e8e7146b6899cfa8fc5fe48ec 2023-03-21 11:58:03,360 DEBUG [StoreCloser-TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/6401b28e4e6a459baeed257a89acb86b to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/6401b28e4e6a459baeed257a89acb86b 2023-03-21 11:58:03,361 DEBUG [StoreCloser-TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/059653d6da58499db49c19ab54892bad to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/059653d6da58499db49c19ab54892bad 2023-03-21 11:58:03,362 DEBUG [StoreCloser-TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/0d4f9c1fb71f44ab8d48af6ea030f1da to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/0d4f9c1fb71f44ab8d48af6ea030f1da 2023-03-21 11:58:03,363 DEBUG [StoreCloser-TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/966966a35deb43208cb7cb1f704fdd06 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/966966a35deb43208cb7cb1f704fdd06 2023-03-21 11:58:03,363 DEBUG [StoreCloser-TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/7245a8d75dab4ef59f461ab0deab300f to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/7245a8d75dab4ef59f461ab0deab300f 2023-03-21 11:58:03,364 DEBUG [StoreCloser-TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/8e3e6a1fac1d44eaa56ce057b6fbffc9 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/8e3e6a1fac1d44eaa56ce057b6fbffc9 2023-03-21 11:58:03,365 DEBUG [StoreCloser-TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/cc95426bd4c94e92bf5a42ddd60409b8 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/cc95426bd4c94e92bf5a42ddd60409b8 2023-03-21 11:58:03,366 DEBUG [StoreCloser-TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/54769945dc7f459d9c2ad0de3690f9e4 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/54769945dc7f459d9c2ad0de3690f9e4 2023-03-21 11:58:03,366 DEBUG [StoreCloser-TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/4d5963850eb840a0996bfe057f5ca4f2 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/4d5963850eb840a0996bfe057f5ca4f2 2023-03-21 11:58:03,367 DEBUG [StoreCloser-TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/29dbd806d71c4ad8879863662b40cf91 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/29dbd806d71c4ad8879863662b40cf91 2023-03-21 11:58:03,368 DEBUG [StoreCloser-TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/6ff5db78cc884b948cf4c91a58b5c3bb to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/6ff5db78cc884b948cf4c91a58b5c3bb 2023-03-21 11:58:03,369 DEBUG [StoreCloser-TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/381d152397b3409699f3688d0fc8048f to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/381d152397b3409699f3688d0fc8048f 2023-03-21 11:58:03,370 DEBUG [StoreCloser-TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/3aeff156e4274296bae04c4b64f94ac8 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/3aeff156e4274296bae04c4b64f94ac8 2023-03-21 11:58:03,370 DEBUG [StoreCloser-TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/016f5e93911c42df83d3f00d4fad3c86 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/016f5e93911c42df83d3f00d4fad3c86 2023-03-21 11:58:03,371 DEBUG [StoreCloser-TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/3e6bd035f78748d780f56036691e83d2 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/3e6bd035f78748d780f56036691e83d2 2023-03-21 11:58:03,374 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/recovered.edits/416.seqid, newMaxSeqId=416, maxSeqId=1 2023-03-21 11:58:03,376 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad. 2023-03-21 11:58:03,376 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 8dbd4df2f9769e1bcf08d64a8aceb9ad: 2023-03-21 11:58:03,378 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 8dbd4df2f9769e1bcf08d64a8aceb9ad 2023-03-21 11:58:03,378 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=8dbd4df2f9769e1bcf08d64a8aceb9ad, regionState=CLOSED 2023-03-21 11:58:03,379 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679399883378"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679399883378"}]},"ts":"1679399883378"} 2023-03-21 11:58:03,381 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=52, resume processing ppid=51 2023-03-21 11:58:03,381 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=52, ppid=51, state=SUCCESS; CloseRegionProcedure 8dbd4df2f9769e1bcf08d64a8aceb9ad, server=jenkins-hbase20.apache.org,34369,1679399732030 in 2.4670 sec 2023-03-21 11:58:03,383 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=51, resume processing ppid=50 2023-03-21 11:58:03,383 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=51, ppid=50, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=8dbd4df2f9769e1bcf08d64a8aceb9ad, UNASSIGN in 2.4690 sec 2023-03-21 11:58:03,383 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679399883383"}]},"ts":"1679399883383"} 2023-03-21 11:58:03,384 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-03-21 11:58:03,386 INFO [PEWorker-2] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-03-21 11:58:03,388 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=50, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 2.4830 sec 2023-03-21 11:58:05,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-03-21 11:58:05,017 INFO [Listener at localhost.localdomain/37475] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 50 completed 2023-03-21 11:58:05,019 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.HMaster$5(2228): Client=jenkins//148.251.75.209 delete TestAcidGuarantees 2023-03-21 11:58:05,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure2.ProcedureExecutor(1029): Stored pid=53, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-03-21 11:58:05,023 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=53, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-21 11:58:05,025 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=53, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-21 11:58:05,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1227): Checking to see if procedure is done pid=53 2023-03-21 11:58:05,031 DEBUG [HFileArchiver-11] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/.tmp/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad 2023-03-21 11:58:05,034 DEBUG [HFileArchiver-11] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/.tmp/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A, FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/.tmp/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B, FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/.tmp/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C, FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/.tmp/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/recovered.edits] 2023-03-21 11:58:05,039 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/.tmp/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/002b492a7a064e5e902276e253c70b27 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/002b492a7a064e5e902276e253c70b27 2023-03-21 11:58:05,041 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/.tmp/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/2f68802f9a96460182d9b2f1085e94ec to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/2f68802f9a96460182d9b2f1085e94ec 2023-03-21 11:58:05,044 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/.tmp/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/8c6848e74fe745ddbac2d979802821eb to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/A/8c6848e74fe745ddbac2d979802821eb 2023-03-21 11:58:05,048 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/.tmp/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/5f7075dd214b45cab2ec623699179f49 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/5f7075dd214b45cab2ec623699179f49 2023-03-21 11:58:05,050 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/.tmp/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/6c4d25d70e584290b7b43af08b98c730 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/6c4d25d70e584290b7b43af08b98c730 2023-03-21 11:58:05,052 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/.tmp/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/706a998f531b49d795e6c6c8918e28ee to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/B/706a998f531b49d795e6c6c8918e28ee 2023-03-21 11:58:05,057 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/.tmp/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/169ee5ef6ecf4cebb4608e796d953e82 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/169ee5ef6ecf4cebb4608e796d953e82 2023-03-21 11:58:05,059 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/.tmp/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/85482d180e0443d6beab1e99126a0295 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/85482d180e0443d6beab1e99126a0295 2023-03-21 11:58:05,062 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/.tmp/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/d358874b64c945839a9ae78ba282bd26 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/C/d358874b64c945839a9ae78ba282bd26 2023-03-21 11:58:05,066 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/.tmp/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/recovered.edits/416.seqid to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad/recovered.edits/416.seqid 2023-03-21 11:58:05,067 DEBUG [HFileArchiver-11] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/.tmp/data/default/TestAcidGuarantees/8dbd4df2f9769e1bcf08d64a8aceb9ad 2023-03-21 11:58:05,067 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-21 11:58:05,070 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=53, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-21 11:58:05,072 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-03-21 11:58:05,074 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-03-21 11:58:05,077 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=53, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-21 11:58:05,077 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-03-21 11:58:05,077 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1679399885077"}]},"ts":"9223372036854775807"} 2023-03-21 11:58:05,079 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-03-21 11:58:05,079 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 8dbd4df2f9769e1bcf08d64a8aceb9ad, NAME => 'TestAcidGuarantees,,1679399857434.8dbd4df2f9769e1bcf08d64a8aceb9ad.', STARTKEY => '', ENDKEY => ''}] 2023-03-21 11:58:05,079 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-03-21 11:58:05,079 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1679399885079"}]},"ts":"9223372036854775807"} 2023-03-21 11:58:05,080 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-03-21 11:58:05,082 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=53, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-21 11:58:05,083 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=53, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 63 msec 2023-03-21 11:58:05,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1227): Checking to see if procedure is done pid=53 2023-03-21 11:58:05,127 INFO [Listener at localhost.localdomain/37475] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 53 completed 2023-03-21 11:58:05,139 INFO [Listener at localhost.localdomain/37475] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testScanAtomicity Thread=228 (was 232), OpenFileDescriptor=550 (was 538) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=383 (was 380) - SystemLoadAverage LEAK? -, ProcessCount=166 (was 167), AvailableMemoryMB=9177 (was 9089) - AvailableMemoryMB LEAK? - 2023-03-21 11:58:05,151 INFO [Listener at localhost.localdomain/37475] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobGetAtomicity Thread=228, OpenFileDescriptor=550, MaxFileDescriptor=60000, SystemLoadAverage=383, ProcessCount=166, AvailableMemoryMB=9176 2023-03-21 11:58:05,153 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-21 11:58:05,153 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.HMaster$4(2112): Client=jenkins//148.251.75.209 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-21 11:58:05,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure2.ProcedureExecutor(1029): Stored pid=54, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-03-21 11:58:05,156 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-03-21 11:58:05,156 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(697): Client=jenkins//148.251.75.209 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 54 2023-03-21 11:58:05,157 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-03-21 11:58:05,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-03-21 11:58:05,158 DEBUG [HFileArchiver-12] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/.tmp/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961 2023-03-21 11:58:05,159 DEBUG [HFileArchiver-12] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/.tmp/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961 empty. 2023-03-21 11:58:05,159 DEBUG [HFileArchiver-12] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/.tmp/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961 2023-03-21 11:58:05,159 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-21 11:58:05,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-03-21 11:58:05,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-03-21 11:58:05,568 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-21 11:58:05,569 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => cc711a8c4e0d80e60aabbf61e79ab961, NAME => 'TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/.tmp 2023-03-21 11:58:05,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-03-21 11:58:05,975 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-21 11:58:05,975 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing cc711a8c4e0d80e60aabbf61e79ab961, disabling compactions & flushes 2023-03-21 11:58:05,976 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:05,976 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:05,976 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. after waiting 0 ms 2023-03-21 11:58:05,976 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:05,976 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:05,976 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for cc711a8c4e0d80e60aabbf61e79ab961: 2023-03-21 11:58:05,981 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-03-21 11:58:05,983 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679399885983"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679399885983"}]},"ts":"1679399885983"} 2023-03-21 11:58:05,985 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-03-21 11:58:05,986 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-03-21 11:58:05,986 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679399885986"}]},"ts":"1679399885986"} 2023-03-21 11:58:05,987 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-03-21 11:58:05,990 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=cc711a8c4e0d80e60aabbf61e79ab961, ASSIGN}] 2023-03-21 11:58:05,992 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=cc711a8c4e0d80e60aabbf61e79ab961, ASSIGN 2023-03-21 11:58:05,993 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=cc711a8c4e0d80e60aabbf61e79ab961, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,34369,1679399732030; forceNewPlan=false, retain=false 2023-03-21 11:58:06,145 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=cc711a8c4e0d80e60aabbf61e79ab961, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:06,146 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679399886145"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1679399886145"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679399886145"}]},"ts":"1679399886145"} 2023-03-21 11:58:06,148 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=56, ppid=55, state=RUNNABLE; OpenRegionProcedure cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030}] 2023-03-21 11:58:06,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-03-21 11:58:06,307 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:06,308 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => cc711a8c4e0d80e60aabbf61e79ab961, NAME => 'TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961.', STARTKEY => '', ENDKEY => ''} 2023-03-21 11:58:06,308 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees cc711a8c4e0d80e60aabbf61e79ab961 2023-03-21 11:58:06,308 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-21 11:58:06,308 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for cc711a8c4e0d80e60aabbf61e79ab961 2023-03-21 11:58:06,309 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for cc711a8c4e0d80e60aabbf61e79ab961 2023-03-21 11:58:06,311 INFO [StoreOpener-cc711a8c4e0d80e60aabbf61e79ab961-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region cc711a8c4e0d80e60aabbf61e79ab961 2023-03-21 11:58:06,314 DEBUG [StoreOpener-cc711a8c4e0d80e60aabbf61e79ab961-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A 2023-03-21 11:58:06,314 DEBUG [StoreOpener-cc711a8c4e0d80e60aabbf61e79ab961-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A 2023-03-21 11:58:06,314 INFO [StoreOpener-cc711a8c4e0d80e60aabbf61e79ab961-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-21 11:58:06,315 INFO [StoreOpener-cc711a8c4e0d80e60aabbf61e79ab961-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region cc711a8c4e0d80e60aabbf61e79ab961 columnFamilyName A 2023-03-21 11:58:06,316 INFO [StoreOpener-cc711a8c4e0d80e60aabbf61e79ab961-1] regionserver.HStore(310): Store=cc711a8c4e0d80e60aabbf61e79ab961/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-21 11:58:06,316 INFO [StoreOpener-cc711a8c4e0d80e60aabbf61e79ab961-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region cc711a8c4e0d80e60aabbf61e79ab961 2023-03-21 11:58:06,318 DEBUG [StoreOpener-cc711a8c4e0d80e60aabbf61e79ab961-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B 2023-03-21 11:58:06,318 DEBUG [StoreOpener-cc711a8c4e0d80e60aabbf61e79ab961-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B 2023-03-21 11:58:06,318 INFO [StoreOpener-cc711a8c4e0d80e60aabbf61e79ab961-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-21 11:58:06,319 INFO [StoreOpener-cc711a8c4e0d80e60aabbf61e79ab961-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region cc711a8c4e0d80e60aabbf61e79ab961 columnFamilyName B 2023-03-21 11:58:06,320 INFO [StoreOpener-cc711a8c4e0d80e60aabbf61e79ab961-1] regionserver.HStore(310): Store=cc711a8c4e0d80e60aabbf61e79ab961/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-21 11:58:06,320 INFO [StoreOpener-cc711a8c4e0d80e60aabbf61e79ab961-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region cc711a8c4e0d80e60aabbf61e79ab961 2023-03-21 11:58:06,322 DEBUG [StoreOpener-cc711a8c4e0d80e60aabbf61e79ab961-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C 2023-03-21 11:58:06,322 DEBUG [StoreOpener-cc711a8c4e0d80e60aabbf61e79ab961-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C 2023-03-21 11:58:06,322 INFO [StoreOpener-cc711a8c4e0d80e60aabbf61e79ab961-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-21 11:58:06,322 INFO [StoreOpener-cc711a8c4e0d80e60aabbf61e79ab961-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region cc711a8c4e0d80e60aabbf61e79ab961 columnFamilyName C 2023-03-21 11:58:06,323 INFO [StoreOpener-cc711a8c4e0d80e60aabbf61e79ab961-1] regionserver.HStore(310): Store=cc711a8c4e0d80e60aabbf61e79ab961/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-21 11:58:06,323 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:06,324 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961 2023-03-21 11:58:06,325 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961 2023-03-21 11:58:06,328 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-21 11:58:06,329 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for cc711a8c4e0d80e60aabbf61e79ab961 2023-03-21 11:58:06,332 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-21 11:58:06,333 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened cc711a8c4e0d80e60aabbf61e79ab961; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61227919, jitterRate=-0.0876329094171524}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-21 11:58:06,333 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for cc711a8c4e0d80e60aabbf61e79ab961: 2023-03-21 11:58:06,334 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961., pid=56, masterSystemTime=1679399886301 2023-03-21 11:58:06,336 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:06,336 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:06,336 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=cc711a8c4e0d80e60aabbf61e79ab961, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:06,337 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679399886336"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1679399886336"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1679399886336"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1679399886336"}]},"ts":"1679399886336"} 2023-03-21 11:58:06,340 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=56, resume processing ppid=55 2023-03-21 11:58:06,340 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=56, ppid=55, state=SUCCESS; OpenRegionProcedure cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 in 190 msec 2023-03-21 11:58:06,342 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=55, resume processing ppid=54 2023-03-21 11:58:06,342 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=55, ppid=54, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=cc711a8c4e0d80e60aabbf61e79ab961, ASSIGN in 350 msec 2023-03-21 11:58:06,343 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-03-21 11:58:06,344 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679399886343"}]},"ts":"1679399886343"} 2023-03-21 11:58:06,345 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-03-21 11:58:06,347 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-03-21 11:58:06,349 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=54, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1950 sec 2023-03-21 11:58:07,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-03-21 11:58:07,268 INFO [Listener at localhost.localdomain/37475] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 54 completed 2023-03-21 11:58:07,269 DEBUG [Listener at localhost.localdomain/37475] zookeeper.ReadOnlyZKClient(139): Connect 0x43e540fb to 127.0.0.1:61761 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-21 11:58:07,281 DEBUG [Listener at localhost.localdomain/37475] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@648a6b82, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-21 11:58:07,283 DEBUG [Listener at localhost.localdomain/37475] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-21 11:58:07,284 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:55100, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-21 11:58:07,286 DEBUG [Listener at localhost.localdomain/37475] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-03-21 11:58:07,289 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:47704, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-03-21 11:58:07,291 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-21 11:58:07,291 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.HMaster$12(2454): Client=jenkins//148.251.75.209 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-21 11:58:07,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure2.ProcedureExecutor(1029): Stored pid=57, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-03-21 11:58:07,703 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-03-21 11:58:07,706 DEBUG [PEWorker-2] util.FSTableDescriptors(519): Deleted hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-21 11:58:07,706 INFO [PEWorker-2] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-03-21 11:58:07,708 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=58, ppid=57, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-03-21 11:58:07,712 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=cc711a8c4e0d80e60aabbf61e79ab961, REOPEN/MOVE}] 2023-03-21 11:58:07,713 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=cc711a8c4e0d80e60aabbf61e79ab961, REOPEN/MOVE 2023-03-21 11:58:07,714 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=cc711a8c4e0d80e60aabbf61e79ab961, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:07,714 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679399887714"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1679399887714"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679399887714"}]},"ts":"1679399887714"} 2023-03-21 11:58:07,715 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=60, ppid=59, state=RUNNABLE; CloseRegionProcedure cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030}] 2023-03-21 11:58:07,868 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close cc711a8c4e0d80e60aabbf61e79ab961 2023-03-21 11:58:07,869 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing cc711a8c4e0d80e60aabbf61e79ab961, disabling compactions & flushes 2023-03-21 11:58:07,870 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:07,870 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:07,870 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. after waiting 0 ms 2023-03-21 11:58:07,870 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:07,876 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-03-21 11:58:07,876 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:07,876 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for cc711a8c4e0d80e60aabbf61e79ab961: 2023-03-21 11:58:07,876 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(3509): Not adding moved region record: cc711a8c4e0d80e60aabbf61e79ab961 to self. 2023-03-21 11:58:07,878 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed cc711a8c4e0d80e60aabbf61e79ab961 2023-03-21 11:58:07,878 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=cc711a8c4e0d80e60aabbf61e79ab961, regionState=CLOSED 2023-03-21 11:58:07,878 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679399887878"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679399887878"}]},"ts":"1679399887878"} 2023-03-21 11:58:07,880 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=60, resume processing ppid=59 2023-03-21 11:58:07,881 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=60, ppid=59, state=SUCCESS; CloseRegionProcedure cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 in 164 msec 2023-03-21 11:58:07,881 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=cc711a8c4e0d80e60aabbf61e79ab961, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase20.apache.org,34369,1679399732030; forceNewPlan=false, retain=true 2023-03-21 11:58:08,032 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=cc711a8c4e0d80e60aabbf61e79ab961, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:08,032 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679399888032"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1679399888032"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679399888032"}]},"ts":"1679399888032"} 2023-03-21 11:58:08,034 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=61, ppid=59, state=RUNNABLE; OpenRegionProcedure cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030}] 2023-03-21 11:58:08,192 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:08,193 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => cc711a8c4e0d80e60aabbf61e79ab961, NAME => 'TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961.', STARTKEY => '', ENDKEY => ''} 2023-03-21 11:58:08,193 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees cc711a8c4e0d80e60aabbf61e79ab961 2023-03-21 11:58:08,193 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-21 11:58:08,194 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for cc711a8c4e0d80e60aabbf61e79ab961 2023-03-21 11:58:08,194 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for cc711a8c4e0d80e60aabbf61e79ab961 2023-03-21 11:58:08,196 INFO [StoreOpener-cc711a8c4e0d80e60aabbf61e79ab961-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region cc711a8c4e0d80e60aabbf61e79ab961 2023-03-21 11:58:08,197 DEBUG [StoreOpener-cc711a8c4e0d80e60aabbf61e79ab961-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A 2023-03-21 11:58:08,197 DEBUG [StoreOpener-cc711a8c4e0d80e60aabbf61e79ab961-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A 2023-03-21 11:58:08,197 INFO [StoreOpener-cc711a8c4e0d80e60aabbf61e79ab961-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-21 11:58:08,198 INFO [StoreOpener-cc711a8c4e0d80e60aabbf61e79ab961-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region cc711a8c4e0d80e60aabbf61e79ab961 columnFamilyName A 2023-03-21 11:58:08,200 INFO [StoreOpener-cc711a8c4e0d80e60aabbf61e79ab961-1] regionserver.HStore(310): Store=cc711a8c4e0d80e60aabbf61e79ab961/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-21 11:58:08,200 INFO [StoreOpener-cc711a8c4e0d80e60aabbf61e79ab961-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region cc711a8c4e0d80e60aabbf61e79ab961 2023-03-21 11:58:08,202 DEBUG [StoreOpener-cc711a8c4e0d80e60aabbf61e79ab961-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B 2023-03-21 11:58:08,202 DEBUG [StoreOpener-cc711a8c4e0d80e60aabbf61e79ab961-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B 2023-03-21 11:58:08,202 INFO [StoreOpener-cc711a8c4e0d80e60aabbf61e79ab961-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-21 11:58:08,203 INFO [StoreOpener-cc711a8c4e0d80e60aabbf61e79ab961-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region cc711a8c4e0d80e60aabbf61e79ab961 columnFamilyName B 2023-03-21 11:58:08,203 INFO [StoreOpener-cc711a8c4e0d80e60aabbf61e79ab961-1] regionserver.HStore(310): Store=cc711a8c4e0d80e60aabbf61e79ab961/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-21 11:58:08,204 INFO [StoreOpener-cc711a8c4e0d80e60aabbf61e79ab961-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region cc711a8c4e0d80e60aabbf61e79ab961 2023-03-21 11:58:08,205 DEBUG [StoreOpener-cc711a8c4e0d80e60aabbf61e79ab961-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C 2023-03-21 11:58:08,205 DEBUG [StoreOpener-cc711a8c4e0d80e60aabbf61e79ab961-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C 2023-03-21 11:58:08,205 INFO [StoreOpener-cc711a8c4e0d80e60aabbf61e79ab961-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-21 11:58:08,205 INFO [StoreOpener-cc711a8c4e0d80e60aabbf61e79ab961-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region cc711a8c4e0d80e60aabbf61e79ab961 columnFamilyName C 2023-03-21 11:58:08,206 INFO [StoreOpener-cc711a8c4e0d80e60aabbf61e79ab961-1] regionserver.HStore(310): Store=cc711a8c4e0d80e60aabbf61e79ab961/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-21 11:58:08,206 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:08,207 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961 2023-03-21 11:58:08,209 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961 2023-03-21 11:58:08,212 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-21 11:58:08,213 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for cc711a8c4e0d80e60aabbf61e79ab961 2023-03-21 11:58:08,214 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened cc711a8c4e0d80e60aabbf61e79ab961; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62933329, jitterRate=-0.06222032010555267}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-21 11:58:08,214 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for cc711a8c4e0d80e60aabbf61e79ab961: 2023-03-21 11:58:08,214 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961., pid=61, masterSystemTime=1679399888187 2023-03-21 11:58:08,216 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:08,216 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:08,216 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=cc711a8c4e0d80e60aabbf61e79ab961, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:08,217 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679399888216"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1679399888216"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1679399888216"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1679399888216"}]},"ts":"1679399888216"} 2023-03-21 11:58:08,219 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=61, resume processing ppid=59 2023-03-21 11:58:08,219 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=61, ppid=59, state=SUCCESS; OpenRegionProcedure cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 in 184 msec 2023-03-21 11:58:08,221 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=59, resume processing ppid=58 2023-03-21 11:58:08,221 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=59, ppid=58, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=cc711a8c4e0d80e60aabbf61e79ab961, REOPEN/MOVE in 507 msec 2023-03-21 11:58:08,225 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=58, resume processing ppid=57 2023-03-21 11:58:08,225 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=58, ppid=57, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 514 msec 2023-03-21 11:58:08,227 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=57, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 935 msec 2023-03-21 11:58:08,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1227): Checking to see if procedure is done pid=57 2023-03-21 11:58:08,230 DEBUG [Listener at localhost.localdomain/37475] zookeeper.ReadOnlyZKClient(139): Connect 0x395f9d24 to 127.0.0.1:61761 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-21 11:58:08,234 DEBUG [Listener at localhost.localdomain/37475] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5054e06c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-21 11:58:08,234 DEBUG [Listener at localhost.localdomain/37475] zookeeper.ReadOnlyZKClient(139): Connect 0x774f1998 to 127.0.0.1:61761 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-21 11:58:08,238 DEBUG [Listener at localhost.localdomain/37475] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@323c0ce8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-21 11:58:08,238 DEBUG [Listener at localhost.localdomain/37475] zookeeper.ReadOnlyZKClient(139): Connect 0x085873d0 to 127.0.0.1:61761 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-21 11:58:08,243 DEBUG [Listener at localhost.localdomain/37475] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@754289a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-21 11:58:08,244 DEBUG [Listener at localhost.localdomain/37475] zookeeper.ReadOnlyZKClient(139): Connect 0x768816e9 to 127.0.0.1:61761 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-21 11:58:08,248 DEBUG [Listener at localhost.localdomain/37475] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3d853d1b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-21 11:58:08,248 DEBUG [Listener at localhost.localdomain/37475] zookeeper.ReadOnlyZKClient(139): Connect 0x27acfde4 to 127.0.0.1:61761 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-21 11:58:08,252 DEBUG [Listener at localhost.localdomain/37475] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@112588a6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-21 11:58:08,253 DEBUG [Listener at localhost.localdomain/37475] zookeeper.ReadOnlyZKClient(139): Connect 0x407f33a2 to 127.0.0.1:61761 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-21 11:58:08,257 DEBUG [Listener at localhost.localdomain/37475] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7d22e3b3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-21 11:58:08,257 DEBUG [Listener at localhost.localdomain/37475] zookeeper.ReadOnlyZKClient(139): Connect 0x7dfddf81 to 127.0.0.1:61761 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-21 11:58:08,266 DEBUG [Listener at localhost.localdomain/37475] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@44a601bf, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-21 11:58:08,266 DEBUG [Listener at localhost.localdomain/37475] zookeeper.ReadOnlyZKClient(139): Connect 0x74adc5bd to 127.0.0.1:61761 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-21 11:58:08,278 DEBUG [Listener at localhost.localdomain/37475] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@67bf0d7e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-21 11:58:08,278 DEBUG [Listener at localhost.localdomain/37475] zookeeper.ReadOnlyZKClient(139): Connect 0x27489815 to 127.0.0.1:61761 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-21 11:58:08,287 DEBUG [Listener at localhost.localdomain/37475] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3d52316c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-21 11:58:08,287 DEBUG [Listener at localhost.localdomain/37475] zookeeper.ReadOnlyZKClient(139): Connect 0x33765ee2 to 127.0.0.1:61761 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-21 11:58:08,295 DEBUG [Listener at localhost.localdomain/37475] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@509de0a9, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-21 11:58:08,308 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:58:08,313 DEBUG [hconnection-0x27bf8931-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-21 11:58:08,313 DEBUG [hconnection-0x1b3168db-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-21 11:58:08,313 DEBUG [hconnection-0x4527562e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-21 11:58:08,313 DEBUG [hconnection-0x31150245-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-21 11:58:08,313 DEBUG [hconnection-0x2aff626f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-21 11:58:08,314 DEBUG [hconnection-0x30e74488-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-21 11:58:08,315 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:55130, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-21 11:58:08,315 DEBUG [hconnection-0x16bd3b37-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-21 11:58:08,316 DEBUG [hconnection-0x2db45121-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-21 11:58:08,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:58:08,315 DEBUG [hconnection-0x287d5471-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-21 11:58:08,317 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:55162, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-21 11:58:08,318 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:55112, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-21 11:58:08,318 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:55110, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-21 11:58:08,318 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:55146, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-21 11:58:08,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:58:08,318 DEBUG [hconnection-0xeeb913c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-21 11:58:08,320 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:55138, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-21 11:58:08,320 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:58:08,321 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:08,321 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:58:08,321 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:58:08,321 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:55190, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-21 11:58:08,321 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:55178, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-21 11:58:08,325 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:55174, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-21 11:58:08,326 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:08,326 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:08,332 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:08,332 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:08,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:08,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:08,332 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:55202, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-21 11:58:08,332 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:08,332 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:58:08,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:08,334 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:08,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:58:08,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:08,334 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:58:08,338 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:58:08,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(9158): Flush requested on cc711a8c4e0d80e60aabbf61e79ab961 2023-03-21 11:58:08,339 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:58:08,339 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing cc711a8c4e0d80e60aabbf61e79ab961 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-21 11:58:08,339 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:58:08,339 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:08,339 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:58:08,340 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:58:08,340 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:58:08,340 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:08,340 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. started... 2023-03-21 11:58:08,340 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. as already flushing 2023-03-21 11:58:08,340 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:08,340 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:58:08,340 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:58:08,340 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:08,340 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:58:08,340 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:58:08,342 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cc711a8c4e0d80e60aabbf61e79ab961, store=A 2023-03-21 11:58:08,342 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:58:08,342 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cc711a8c4e0d80e60aabbf61e79ab961, store=B 2023-03-21 11:58:08,342 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:58:08,342 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cc711a8c4e0d80e60aabbf61e79ab961, store=C 2023-03-21 11:58:08,342 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:58:08,344 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:08,344 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:08,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:08,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:08,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:08,344 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:08,344 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:58:08,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:08,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:08,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:08,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:08,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:08,346 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:58:08,346 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@54fae207[Count = 0] remaining members to acquire global barrier 2023-03-21 11:58:08,346 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:58:08,346 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:08,347 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:08,347 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:58:08,349 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:08,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:08,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:08,350 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:58:08,350 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:58:08,350 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:58:08,350 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:58:08,350 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:08,350 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:08,351 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:58:08,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:08,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:08,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:08,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:08,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:08,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:08,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:08,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:08,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:08,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:08,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:58:08,353 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:58:08,353 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:58:08,353 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:58:08,353 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:58:08,353 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:58:08,353 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:58:08,354 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:08,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:08,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:08,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:08,354 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:08,354 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:08,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:08,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:08,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:08,355 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:08,355 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:08,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55174 deadline: 1679399948353, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:08,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55178 deadline: 1679399948353, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:08,355 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:08,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55146 deadline: 1679399948353, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:08,355 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:08,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55130 deadline: 1679399948354, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:08,356 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:08,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55112 deadline: 1679399948355, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:08,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:08,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:08,358 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:08,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:08,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:08,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:08,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:08,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:08,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:08,359 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:08,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:08,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:08,365 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:08,365 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:08,365 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:08,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:08,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:08,365 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:08,365 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:58:08,365 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:58:08,365 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:08,365 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:08,365 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:08,365 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:08,365 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:08,365 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:08,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:08,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:08,366 DEBUG [Thread-3389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:58:08,366 DEBUG [Thread-3389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:58:08,457 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:08,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55178 deadline: 1679399948456, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:08,457 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:08,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55146 deadline: 1679399948456, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:08,457 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:08,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55130 deadline: 1679399948456, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:08,457 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:08,458 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:08,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55174 deadline: 1679399948456, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:08,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55112 deadline: 1679399948456, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:08,466 DEBUG [Thread-3389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:58:08,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:58:08,467 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:58:08,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:58:08,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:58:08,468 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:58:08,468 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:08,468 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:58:08,468 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:58:08,468 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:08,468 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:08,469 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:08,469 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:08,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:08,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:08,469 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:08,469 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:58:08,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:08,470 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:08,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:58:08,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:08,470 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:58:08,470 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:58:08,470 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:58:08,470 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:58:08,470 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:08,470 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:58:08,470 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:58:08,470 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:58:08,470 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:08,470 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. started... 2023-03-21 11:58:08,470 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. as already flushing 2023-03-21 11:58:08,470 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:08,471 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:58:08,471 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:58:08,471 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:08,471 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:58:08,471 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:58:08,471 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:08,471 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:08,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:08,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:08,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:08,471 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:08,471 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:58:08,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:08,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:08,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:08,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:08,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:08,472 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:58:08,472 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50dcebd9[Count = 0] remaining members to acquire global barrier 2023-03-21 11:58:08,472 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:58:08,472 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:08,473 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:08,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:08,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:08,473 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:58:08,473 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:08,473 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:58:08,473 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:58:08,473 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:58:08,474 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:08,474 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:58:08,474 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:08,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:08,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:08,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:08,474 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:58:08,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:08,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:08,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:08,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:08,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:08,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:08,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:08,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:58:08,475 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:58:08,475 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:58:08,476 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:58:08,476 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:58:08,476 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:58:08,476 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:58:08,477 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:08,477 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:08,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:08,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:08,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:08,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:08,477 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:08,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:08,477 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:08,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:08,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:08,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:08,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:08,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:08,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:08,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:08,478 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:08,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:08,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:08,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:08,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:08,480 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:08,480 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:08,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:08,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:08,480 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:08,480 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:08,480 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:58:08,480 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:58:08,480 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:08,480 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:08,480 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:08,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:08,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:08,480 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:08,480 DEBUG [Thread-3389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:58:08,480 DEBUG [Thread-3389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:58:08,480 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:08,480 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:08,581 DEBUG [Thread-3389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:58:08,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:58:08,582 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:58:08,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:58:08,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:58:08,583 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:58:08,583 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:08,583 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:58:08,583 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:58:08,583 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:08,583 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:08,584 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:08,584 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:08,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:08,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:08,584 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:08,584 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:58:08,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:08,585 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:08,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:58:08,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:08,585 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:58:08,585 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:58:08,585 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:58:08,585 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:58:08,585 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:08,585 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:58:08,585 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:58:08,585 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:58:08,585 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:08,586 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. started... 2023-03-21 11:58:08,586 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. as already flushing 2023-03-21 11:58:08,586 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:08,586 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:58:08,586 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:58:08,586 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:08,586 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:58:08,586 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:58:08,586 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:08,586 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:08,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:08,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:08,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:08,587 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:08,587 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:58:08,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:08,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:08,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:08,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:08,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:08,588 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:58:08,588 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@25040b01[Count = 0] remaining members to acquire global barrier 2023-03-21 11:58:08,588 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:58:08,588 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:08,588 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:08,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:08,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:08,588 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:08,588 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:58:08,588 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:58:08,588 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:58:08,588 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:58:08,589 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:58:08,589 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:08,589 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:08,589 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:58:08,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:08,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:08,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:08,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:08,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:08,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:08,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:08,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:08,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:08,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:08,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:58:08,591 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:58:08,591 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:58:08,591 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:58:08,591 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:58:08,591 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:58:08,591 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:58:08,592 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:08,592 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:08,592 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:08,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:08,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:08,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:08,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:08,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:08,592 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:08,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:08,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:08,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:08,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:08,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:08,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:08,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:08,593 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:08,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:08,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:08,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:08,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:08,594 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:08,594 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:08,594 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:08,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:08,594 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:08,594 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:08,595 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:58:08,594 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:58:08,595 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:58:08,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:08,594 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:08,595 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:08,595 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:08,595 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:08,595 DEBUG [Thread-3389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:58:08,595 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:08,595 DEBUG [Thread-3389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:58:08,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:08,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:08,660 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:08,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55146 deadline: 1679399948658, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:08,660 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:08,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55174 deadline: 1679399948658, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:08,660 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:08,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55178 deadline: 1679399948659, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:08,661 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:08,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55130 deadline: 1679399948659, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:08,661 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:08,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55112 deadline: 1679399948659, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:08,695 DEBUG [Thread-3389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:58:08,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:58:08,697 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:58:08,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:58:08,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:58:08,698 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:58:08,698 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:08,698 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:58:08,698 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:58:08,698 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:08,698 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:08,699 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:08,699 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:08,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:08,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:08,699 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:08,699 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:58:08,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:08,699 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:08,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:58:08,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:08,700 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:58:08,700 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:58:08,700 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:58:08,700 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:58:08,700 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:08,700 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:58:08,700 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:58:08,700 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:58:08,700 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:08,700 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. started... 2023-03-21 11:58:08,700 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. as already flushing 2023-03-21 11:58:08,700 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:08,700 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:58:08,700 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:58:08,700 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:08,700 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:58:08,700 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:58:08,701 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:08,701 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:08,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:08,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:08,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:08,701 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:08,701 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:58:08,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:08,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:08,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:08,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:08,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:08,702 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:58:08,702 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f1b164f[Count = 0] remaining members to acquire global barrier 2023-03-21 11:58:08,702 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:58:08,702 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:08,703 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:08,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:08,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:08,703 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:58:08,703 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:58:08,703 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:08,703 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:58:08,703 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:58:08,704 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:08,704 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:58:08,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:08,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:08,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:08,704 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:08,704 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:58:08,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:08,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:08,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:08,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:08,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:08,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:08,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:08,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:58:08,705 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:58:08,705 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:58:08,705 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:58:08,705 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:58:08,705 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:58:08,705 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:58:08,706 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:08,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:08,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:08,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:08,706 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:08,706 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:08,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:08,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:08,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:08,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:08,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:08,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:08,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:08,707 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:08,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:08,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:08,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:08,708 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:08,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:08,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:08,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:08,709 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:08,709 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:08,709 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:08,709 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:08,709 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:58:08,710 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:58:08,709 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:08,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:08,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:08,709 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:08,710 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:08,710 DEBUG [Thread-3389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:58:08,710 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:08,710 DEBUG [Thread-3389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:58:08,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:08,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:08,710 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:08,710 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:08,760 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303212ee8bfb8a437455d87312f2745e9365f to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303212ee8bfb8a437455d87312f2745e9365f 2023-03-21 11:58:08,810 DEBUG [Thread-3389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:58:08,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:58:08,811 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:58:08,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:58:08,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:58:08,812 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:58:08,812 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:08,813 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:58:08,813 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:58:08,813 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:08,813 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:08,813 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:08,813 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:08,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:08,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:08,814 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:08,814 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:58:08,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:08,814 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:08,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:58:08,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:08,814 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:58:08,814 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:58:08,815 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:58:08,815 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:58:08,815 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:08,815 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:58:08,815 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:58:08,815 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:58:08,815 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:08,815 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. started... 2023-03-21 11:58:08,815 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. as already flushing 2023-03-21 11:58:08,815 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:08,815 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:58:08,815 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:58:08,815 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:08,815 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:58:08,815 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:58:08,816 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:08,816 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:08,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:08,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:08,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:08,816 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:08,816 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:58:08,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:08,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:08,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:08,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:08,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:08,817 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:58:08,817 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c723d71[Count = 0] remaining members to acquire global barrier 2023-03-21 11:58:08,817 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:58:08,817 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:08,818 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:08,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:08,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:08,818 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:58:08,818 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:58:08,818 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:58:08,819 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:08,819 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:58:08,819 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:08,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:08,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:08,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:08,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:08,820 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:58:08,820 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:08,820 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:58:08,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:08,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:08,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:08,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:08,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:08,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:08,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:58:08,821 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:58:08,821 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:58:08,821 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:58:08,821 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:58:08,821 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:58:08,821 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:58:08,822 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:08,822 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:08,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:08,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:08,822 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:08,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:08,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:08,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:08,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:08,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:08,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:08,823 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:08,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:08,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:08,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:08,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:08,824 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:08,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:08,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:08,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:08,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:08,826 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:08,826 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:08,826 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:08,826 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:08,826 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:08,826 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:08,826 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:08,826 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:08,826 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:08,826 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:58:08,826 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:58:08,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:08,826 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:58:08,826 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:08,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:08,826 DEBUG [Thread-3389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:58:08,826 DEBUG [Thread-3389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:58:08,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:08,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:08,900 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-03-21 11:58:08,926 DEBUG [Thread-3389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:58:08,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:58:08,928 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:58:08,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:58:08,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:58:08,938 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:58:08,938 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:08,938 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:58:08,938 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:58:08,939 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:08,939 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:08,939 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:08,939 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:08,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:08,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:08,940 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:08,940 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:58:08,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:08,940 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:08,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:58:08,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:08,940 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:58:08,941 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:58:08,941 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:58:08,941 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:58:08,941 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:08,941 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:58:08,942 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:58:08,942 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:58:08,942 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:08,942 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. started... 2023-03-21 11:58:08,942 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. as already flushing 2023-03-21 11:58:08,942 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:08,942 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:58:08,943 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:58:08,943 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:08,943 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:58:08,943 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:58:08,944 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:08,944 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:08,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:08,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:08,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:08,944 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:08,944 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:58:08,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:08,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:08,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:08,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:08,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:08,945 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:58:08,945 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6b479705[Count = 0] remaining members to acquire global barrier 2023-03-21 11:58:08,945 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:58:08,945 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:08,946 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:08,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:08,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:08,946 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:58:08,946 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:58:08,946 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:58:08,946 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:08,946 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:58:08,946 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:08,946 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:58:08,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:08,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:08,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:08,946 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:08,947 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:58:08,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:08,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:08,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:08,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:08,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:08,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:08,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:08,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:58:08,948 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:58:08,948 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:58:08,948 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:58:08,948 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:58:08,948 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:58:08,948 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:58:08,949 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:08,949 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:08,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:08,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:08,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:08,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:08,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:08,949 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:08,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:08,949 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:08,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:08,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:08,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:08,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:08,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:08,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:08,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:08,950 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:08,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:08,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:08,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:08,952 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:08,952 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:08,952 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:08,952 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:08,952 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:58:08,952 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:58:08,952 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:08,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:08,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:08,952 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:08,952 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:08,952 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:08,952 DEBUG [Thread-3389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:58:08,952 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:08,952 DEBUG [Thread-3389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:58:08,952 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:08,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:08,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:08,962 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:08,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55178 deadline: 1679399948961, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:08,963 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:08,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55146 deadline: 1679399948961, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:08,963 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:08,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55112 deadline: 1679399948961, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:08,963 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:08,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55174 deadline: 1679399948962, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:08,964 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:08,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55130 deadline: 1679399948962, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:09,052 DEBUG [Thread-3389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:58:09,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:58:09,054 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:58:09,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:58:09,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:58:09,055 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:58:09,055 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:09,055 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:58:09,055 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:58:09,055 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:09,055 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:09,056 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:09,056 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:09,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:09,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:09,056 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:09,056 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:58:09,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:09,057 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:09,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:58:09,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:09,057 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:58:09,057 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:58:09,057 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:58:09,057 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:58:09,057 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:09,057 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:58:09,057 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:58:09,057 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:09,057 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:58:09,057 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. started... 2023-03-21 11:58:09,058 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. as already flushing 2023-03-21 11:58:09,058 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:09,058 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:58:09,058 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:58:09,058 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:09,058 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:58:09,058 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:58:09,059 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:09,059 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:09,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:09,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:09,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:09,059 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:09,059 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:58:09,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:09,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:09,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:09,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:09,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:09,060 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:58:09,060 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e9ed56f[Count = 0] remaining members to acquire global barrier 2023-03-21 11:58:09,060 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:58:09,060 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:09,060 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:09,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:09,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:09,060 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:58:09,061 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:58:09,060 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:09,061 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:58:09,061 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:58:09,061 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:09,061 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:58:09,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:09,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:09,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:09,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:09,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:09,061 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:09,062 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:58:09,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:09,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:09,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:09,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:09,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:09,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:58:09,063 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:58:09,063 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:58:09,063 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:58:09,063 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:58:09,063 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:58:09,063 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:58:09,064 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:09,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:09,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:09,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:09,064 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:09,064 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:09,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:09,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:09,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:09,065 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:09,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:09,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:09,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:09,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:09,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:09,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:09,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:09,066 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:09,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:09,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:09,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:09,067 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:09,067 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:09,067 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:09,067 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:09,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:09,067 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:58:09,068 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:58:09,067 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:09,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:09,067 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:09,068 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:09,068 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:09,068 DEBUG [Thread-3389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:58:09,068 DEBUG [Thread-3389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:58:09,068 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:09,068 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:09,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:09,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:09,165 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=17, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/A/9bee3a786dea4a26bf8ed4cb00ead6a8 2023-03-21 11:58:09,168 DEBUG [Thread-3389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:58:09,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:58:09,170 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:58:09,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:58:09,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:58:09,172 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:58:09,172 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:09,172 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:58:09,172 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:58:09,172 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:09,172 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:09,173 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:09,173 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:09,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:09,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:09,173 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:09,173 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:58:09,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:09,174 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:09,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:58:09,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:09,174 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:58:09,174 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:58:09,175 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:58:09,175 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:58:09,175 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:09,175 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:58:09,175 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=17 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/B/ab7aacb9bff24b1aa02f840ba3d309ed 2023-03-21 11:58:09,175 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:58:09,175 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:58:09,175 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:09,175 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. started... 2023-03-21 11:58:09,175 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. as already flushing 2023-03-21 11:58:09,175 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:09,175 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:58:09,175 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:58:09,175 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:09,175 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:58:09,175 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:58:09,176 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:09,176 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:09,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:09,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:09,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:09,177 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:09,177 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:58:09,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:09,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:09,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:09,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:09,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:09,178 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:58:09,178 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69301f9d[Count = 0] remaining members to acquire global barrier 2023-03-21 11:58:09,178 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:58:09,178 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:09,179 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:09,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:09,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:09,179 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:58:09,179 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:58:09,179 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:09,179 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:58:09,179 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:58:09,180 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:09,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:09,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:09,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:09,180 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:58:09,180 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:09,180 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:58:09,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:09,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:09,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:09,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:09,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:09,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:09,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:09,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:58:09,182 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:58:09,182 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:58:09,182 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:58:09,182 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:58:09,182 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:58:09,182 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:58:09,183 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:09,183 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:09,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:09,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:09,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:09,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:09,183 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:09,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:09,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:09,184 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:09,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:09,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:09,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:09,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:09,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:09,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:09,184 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:09,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:09,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:09,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:09,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:09,186 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:09,186 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:09,186 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:09,186 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:09,186 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:58:09,187 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:09,187 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:58:09,187 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:09,186 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:09,187 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:09,187 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:09,187 DEBUG [Thread-3389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:58:09,187 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:09,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:09,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:09,187 DEBUG [Thread-3389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:58:09,187 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:58:09,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:09,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:09,287 DEBUG [Thread-3389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:58:09,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:58:09,289 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:58:09,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:58:09,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:58:09,290 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:58:09,290 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:09,290 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:58:09,290 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:58:09,290 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:09,290 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:09,291 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:09,291 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:09,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:09,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:09,291 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:09,291 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:58:09,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:09,292 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:09,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:58:09,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:09,292 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:58:09,292 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:58:09,292 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:58:09,292 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:09,292 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:58:09,292 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:58:09,292 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:58:09,292 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:09,292 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. started... 2023-03-21 11:58:09,292 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. as already flushing 2023-03-21 11:58:09,292 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:09,293 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:58:09,293 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:58:09,293 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:09,293 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:58:09,293 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:58:09,293 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:09,293 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:09,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:09,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:09,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:09,294 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:09,294 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:58:09,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:09,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:09,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:09,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:09,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:09,295 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:58:09,295 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@24643e7f[Count = 0] remaining members to acquire global barrier 2023-03-21 11:58:09,295 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:58:09,295 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:09,295 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:09,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:09,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:09,295 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:58:09,296 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:09,296 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:58:09,296 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:58:09,296 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:58:09,296 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:58:09,296 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:09,296 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:09,296 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:58:09,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:09,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:09,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:09,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:09,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:09,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:09,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:09,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:09,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:09,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:09,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:58:09,298 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:58:09,298 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:58:09,298 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:58:09,298 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:58:09,299 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:58:09,299 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:58:09,299 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:09,299 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:09,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:09,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:09,299 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:09,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:09,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:09,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:09,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:09,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:09,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:09,300 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:09,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:09,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:09,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:09,301 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:09,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:09,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:09,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:09,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:09,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:09,302 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:09,303 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:09,303 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:09,303 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:09,303 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:09,303 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:58:09,303 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:58:09,303 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:09,303 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:58:09,303 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:09,303 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:09,303 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:09,303 DEBUG [Thread-3389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:58:09,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:09,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:09,303 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:09,303 DEBUG [Thread-3389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:58:09,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:09,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:09,403 DEBUG [Thread-3389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:58:09,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:58:09,405 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:58:09,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:58:09,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:58:09,406 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:58:09,406 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:09,406 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:58:09,406 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:58:09,406 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:09,406 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:09,407 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:09,407 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:09,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:09,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:09,407 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:09,407 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:58:09,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:09,407 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:09,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:58:09,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:09,407 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:58:09,407 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:58:09,407 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:58:09,407 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:58:09,407 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:09,407 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:58:09,408 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:58:09,408 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:58:09,408 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:09,408 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. started... 2023-03-21 11:58:09,408 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. as already flushing 2023-03-21 11:58:09,408 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:09,408 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:58:09,408 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:58:09,408 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:09,408 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:58:09,408 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:58:09,409 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:09,409 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:09,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:09,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:09,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:09,409 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:09,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:09,409 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:58:09,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:09,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:09,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:09,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:09,410 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:58:09,410 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@734c6a49[Count = 0] remaining members to acquire global barrier 2023-03-21 11:58:09,410 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:58:09,410 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:09,410 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:09,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:09,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:09,410 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:58:09,410 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:58:09,411 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:58:09,411 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:09,411 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:58:09,411 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:58:09,411 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:09,411 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:09,411 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:58:09,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:09,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:09,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:09,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:09,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:09,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:09,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:09,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:09,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:09,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:09,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:58:09,413 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:58:09,413 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:58:09,413 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:58:09,413 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:58:09,413 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:58:09,413 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:58:09,414 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:09,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:09,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:09,414 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:09,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:09,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:09,414 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:09,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:09,414 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:09,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:09,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:09,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:09,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:09,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:09,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:09,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:09,415 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:09,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:09,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:09,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:09,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:09,417 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:09,417 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:09,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:09,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:09,417 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:58:09,417 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:09,417 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:58:09,417 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:58:09,417 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:09,417 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:09,417 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:09,417 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:09,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:09,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:09,417 DEBUG [Thread-3389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:58:09,417 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:09,417 DEBUG [Thread-3389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:58:09,417 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:09,417 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:09,466 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:09,466 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:09,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55112 deadline: 1679399949464, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:09,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55178 deadline: 1679399949464, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:09,466 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:09,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55146 deadline: 1679399949465, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:09,466 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:09,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55174 deadline: 1679399949465, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:09,467 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:09,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55130 deadline: 1679399949465, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:09,518 DEBUG [Thread-3389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:58:09,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:58:09,519 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:58:09,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:58:09,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:58:09,520 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:58:09,520 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:09,520 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:58:09,520 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:58:09,520 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:09,520 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:09,521 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:09,521 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:09,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:09,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:09,521 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:09,521 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:58:09,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:09,521 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:09,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:58:09,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:09,522 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:58:09,523 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:58:09,523 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:58:09,523 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:58:09,523 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:09,523 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:58:09,523 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:58:09,523 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:58:09,523 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:09,523 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. started... 2023-03-21 11:58:09,523 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. as already flushing 2023-03-21 11:58:09,523 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:09,523 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:58:09,523 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:58:09,523 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:09,523 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:58:09,523 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:58:09,524 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:09,524 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:09,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:09,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:09,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:09,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:09,525 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:09,525 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:58:09,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:09,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:09,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:09,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:09,526 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:58:09,526 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6be1fce2[Count = 0] remaining members to acquire global barrier 2023-03-21 11:58:09,526 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:58:09,526 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:09,526 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:09,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:09,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:09,526 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:58:09,526 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:58:09,526 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:58:09,527 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:09,527 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:58:09,527 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:09,527 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:58:09,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:09,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:09,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:09,527 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:09,527 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:58:09,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:09,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:09,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:09,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:09,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:09,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:09,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:09,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:58:09,529 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:58:09,529 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:58:09,529 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:58:09,529 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:58:09,529 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:58:09,529 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:58:09,530 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:09,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:09,530 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:09,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:09,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:09,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:09,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:09,530 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:09,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:09,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:09,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:09,530 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:09,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:09,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:09,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:09,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:09,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:09,532 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:09,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:09,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:09,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:09,533 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:09,533 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:09,533 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:09,533 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:09,533 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:09,533 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:09,533 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:09,533 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:09,533 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:09,533 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:58:09,533 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:58:09,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:09,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:09,533 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:09,534 DEBUG [Thread-3389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:58:09,534 DEBUG [Thread-3389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:58:09,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:09,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:09,583 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=17 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/C/ae2c3046634b4a2dbf982036bddb03d9 2023-03-21 11:58:09,586 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/A/9bee3a786dea4a26bf8ed4cb00ead6a8 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/9bee3a786dea4a26bf8ed4cb00ead6a8 2023-03-21 11:58:09,588 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/9bee3a786dea4a26bf8ed4cb00ead6a8, entries=150, sequenceid=17, filesize=25.1 K 2023-03-21 11:58:09,589 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/B/ab7aacb9bff24b1aa02f840ba3d309ed as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/ab7aacb9bff24b1aa02f840ba3d309ed 2023-03-21 11:58:09,592 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/ab7aacb9bff24b1aa02f840ba3d309ed, entries=150, sequenceid=17, filesize=11.6 K 2023-03-21 11:58:09,592 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/C/ae2c3046634b4a2dbf982036bddb03d9 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/ae2c3046634b4a2dbf982036bddb03d9 2023-03-21 11:58:09,595 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/ae2c3046634b4a2dbf982036bddb03d9, entries=150, sequenceid=17, filesize=11.6 K 2023-03-21 11:58:09,595 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=114.05 KB/116790 for cc711a8c4e0d80e60aabbf61e79ab961 in 1256ms, sequenceid=17, compaction requested=false 2023-03-21 11:58:09,595 DEBUG [MemStoreFlusher.0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-03-21 11:58:09,596 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for cc711a8c4e0d80e60aabbf61e79ab961: 2023-03-21 11:58:09,634 DEBUG [Thread-3389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:58:09,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:58:09,635 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:58:09,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:58:09,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:58:09,636 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:58:09,636 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:09,636 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:58:09,636 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:58:09,637 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:09,637 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:09,638 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:09,638 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:09,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:09,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:09,638 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:09,638 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:58:09,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:09,639 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:09,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:58:09,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:09,639 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:58:09,639 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:58:09,639 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:58:09,639 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:58:09,639 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:09,639 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:58:09,640 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:58:09,640 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:58:09,640 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:09,640 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. started... 2023-03-21 11:58:09,640 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing cc711a8c4e0d80e60aabbf61e79ab961 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-21 11:58:09,640 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cc711a8c4e0d80e60aabbf61e79ab961, store=A 2023-03-21 11:58:09,640 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:58:09,640 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cc711a8c4e0d80e60aabbf61e79ab961, store=B 2023-03-21 11:58:09,640 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:58:09,640 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cc711a8c4e0d80e60aabbf61e79ab961, store=C 2023-03-21 11:58:09,640 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:58:09,654 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230321823b6446b9ed4d6b9d45b1e535bdf3b5 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230321823b6446b9ed4d6b9d45b1e535bdf3b5 2023-03-21 11:58:10,058 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=37, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/A/9842576d1c3f4d029f7e0d4bd1a2b1c1 2023-03-21 11:58:10,066 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=37 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/B/709cf3f129ee491ca2dce049878626f7 2023-03-21 11:58:10,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(9158): Flush requested on cc711a8c4e0d80e60aabbf61e79ab961 2023-03-21 11:58:10,468 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. as already flushing 2023-03-21 11:58:10,473 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=37 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/C/fa5e691f8c3b4f18b9a12e51e5392da5 2023-03-21 11:58:10,478 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/A/9842576d1c3f4d029f7e0d4bd1a2b1c1 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/9842576d1c3f4d029f7e0d4bd1a2b1c1 2023-03-21 11:58:10,478 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:10,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55112 deadline: 1679399950476, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:10,479 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:10,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55130 deadline: 1679399950477, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:10,481 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:10,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55174 deadline: 1679399950477, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:10,481 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:10,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55146 deadline: 1679399950478, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:10,481 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:10,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55178 deadline: 1679399950479, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:10,482 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/9842576d1c3f4d029f7e0d4bd1a2b1c1, entries=150, sequenceid=37, filesize=25.1 K 2023-03-21 11:58:10,482 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/B/709cf3f129ee491ca2dce049878626f7 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/709cf3f129ee491ca2dce049878626f7 2023-03-21 11:58:10,485 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/709cf3f129ee491ca2dce049878626f7, entries=150, sequenceid=37, filesize=11.6 K 2023-03-21 11:58:10,486 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/C/fa5e691f8c3b4f18b9a12e51e5392da5 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/fa5e691f8c3b4f18b9a12e51e5392da5 2023-03-21 11:58:10,489 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/fa5e691f8c3b4f18b9a12e51e5392da5, entries=150, sequenceid=37, filesize=11.6 K 2023-03-21 11:58:10,490 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for cc711a8c4e0d80e60aabbf61e79ab961 in 850ms, sequenceid=37, compaction requested=false 2023-03-21 11:58:10,497 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for cc711a8c4e0d80e60aabbf61e79ab961: 2023-03-21 11:58:10,497 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:10,497 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:58:10,497 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:58:10,497 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:10,497 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:58:10,497 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:58:10,498 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:10,498 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:10,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:10,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:10,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:10,498 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:10,498 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:58:10,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:10,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:10,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:10,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:10,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:10,499 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:58:10,499 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c4daf4[Count = 0] remaining members to acquire global barrier 2023-03-21 11:58:10,499 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:58:10,500 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:10,500 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:10,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:10,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:10,500 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:58:10,500 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:58:10,500 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:58:10,500 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:10,500 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:58:10,501 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:58:10,501 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:10,501 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:58:10,501 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:10,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:10,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:10,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:10,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:10,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:10,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:10,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:10,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:10,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:10,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:10,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:58:10,503 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:58:10,504 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:58:10,504 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:58:10,504 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:58:10,504 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:58:10,504 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:58:10,505 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:10,505 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:10,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:10,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:10,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:10,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:10,505 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:10,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:10,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:10,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:10,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:10,505 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:10,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:10,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:10,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:10,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:10,506 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:10,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:10,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:10,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:10,508 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:10,508 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:10,508 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:10,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:10,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:10,508 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:10,508 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:10,508 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:10,508 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:10,508 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:10,508 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:10,508 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:58:10,508 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:58:10,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:10,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:10,508 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:10,509 DEBUG [Thread-3389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:58:10,509 DEBUG [Thread-3389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:58:10,509 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:58:10,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(9158): Flush requested on cc711a8c4e0d80e60aabbf61e79ab961 2023-03-21 11:58:10,582 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing cc711a8c4e0d80e60aabbf61e79ab961 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-21 11:58:10,582 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cc711a8c4e0d80e60aabbf61e79ab961, store=A 2023-03-21 11:58:10,582 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:58:10,582 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cc711a8c4e0d80e60aabbf61e79ab961, store=B 2023-03-21 11:58:10,582 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:58:10,582 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cc711a8c4e0d80e60aabbf61e79ab961, store=C 2023-03-21 11:58:10,583 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:58:10,592 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303212cf0b47938da4c198c78bd2cd118f807 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303212cf0b47938da4c198c78bd2cd118f807 2023-03-21 11:58:10,596 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=51, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/A/4240e15c12b04d3e81f24438d9a29734 2023-03-21 11:58:10,598 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:10,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55130 deadline: 1679399950595, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:10,599 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:10,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55174 deadline: 1679399950596, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:10,601 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:10,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55146 deadline: 1679399950599, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:10,601 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:10,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55178 deadline: 1679399950599, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:10,601 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:10,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55112 deadline: 1679399950599, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:10,605 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=51 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/B/3d19e94189124e4cbc4e47a2465b3629 2023-03-21 11:58:10,609 DEBUG [Thread-3389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:58:10,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:58:10,611 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:58:10,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:58:10,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:58:10,618 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:58:10,618 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:10,618 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:58:10,618 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:58:10,618 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:10,618 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:10,619 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:10,619 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:10,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:10,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:10,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:10,620 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:10,620 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:58:10,620 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:10,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:58:10,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:10,620 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:58:10,621 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:58:10,621 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:58:10,621 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:58:10,621 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:10,621 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:58:10,621 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:58:10,621 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:58:10,621 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:10,621 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. started... 2023-03-21 11:58:10,621 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. as already flushing 2023-03-21 11:58:10,621 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:10,621 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:58:10,621 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:58:10,621 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:10,621 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:58:10,621 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:58:10,622 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:10,622 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:10,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:10,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:10,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:10,622 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:10,622 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:58:10,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:10,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:10,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:10,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:10,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:10,623 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:58:10,623 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7d68b15[Count = 0] remaining members to acquire global barrier 2023-03-21 11:58:10,623 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:58:10,623 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:10,624 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:10,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:10,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:10,624 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:58:10,624 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:58:10,624 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:10,624 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:58:10,624 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:58:10,625 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:10,625 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:58:10,625 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:10,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:10,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:10,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:10,625 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:58:10,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:10,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:10,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:10,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:10,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:10,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:10,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:10,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:58:10,627 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:58:10,627 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:58:10,627 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:58:10,627 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:58:10,627 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:58:10,627 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:58:10,628 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:10,628 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:10,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:10,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:10,629 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:10,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:10,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:10,629 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:10,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:10,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:10,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:10,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:10,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:10,630 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:10,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:10,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:10,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:10,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:10,631 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:10,631 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:10,632 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:10,632 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:10,632 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:10,632 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:58:10,632 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:58:10,632 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:10,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:10,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:10,632 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:10,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:10,632 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:10,632 DEBUG [Thread-3389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:58:10,632 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:10,632 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:10,632 DEBUG [Thread-3389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:58:10,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:10,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:10,632 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:58:10,701 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:10,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55130 deadline: 1679399950699, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:10,701 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:10,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55174 deadline: 1679399950699, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:10,703 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:10,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55146 deadline: 1679399950702, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:10,703 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:10,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55178 deadline: 1679399950702, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:10,703 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:10,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55112 deadline: 1679399950702, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:10,732 DEBUG [Thread-3389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:58:10,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:58:10,734 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:58:10,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:58:10,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:58:10,735 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:58:10,735 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:10,735 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:58:10,735 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:58:10,735 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:10,735 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:10,736 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:10,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:10,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:10,736 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:10,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:10,736 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:10,736 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:58:10,736 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:10,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:58:10,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:10,737 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:58:10,737 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:58:10,737 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:58:10,737 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:58:10,737 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:10,737 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:58:10,737 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:58:10,737 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:58:10,737 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:10,737 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. started... 2023-03-21 11:58:10,737 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. as already flushing 2023-03-21 11:58:10,737 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:10,737 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:58:10,738 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:58:10,738 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:10,738 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:58:10,738 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:58:10,738 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:10,738 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:10,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:10,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:10,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:10,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:10,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:10,739 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:10,739 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:58:10,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:10,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:10,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:10,740 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:58:10,740 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@17d8a784[Count = 0] remaining members to acquire global barrier 2023-03-21 11:58:10,740 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:58:10,740 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:10,740 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:10,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:10,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:10,740 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:58:10,740 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:58:10,740 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:10,740 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:58:10,740 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:58:10,741 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:10,741 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:58:10,741 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:10,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:10,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:10,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:10,741 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:58:10,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:10,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:10,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:10,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:10,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:10,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:10,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:10,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:58:10,743 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:58:10,743 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:58:10,743 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:58:10,743 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:58:10,743 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:58:10,743 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:58:10,743 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:10,743 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:10,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:10,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:10,743 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:10,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:10,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:10,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:10,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:10,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:10,744 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:10,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:10,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:10,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:10,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:10,745 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:10,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:10,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:10,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:10,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:10,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:10,746 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:10,746 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:10,746 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:10,746 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:10,746 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:10,746 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:58:10,746 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:58:10,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:10,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:10,746 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:10,746 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:58:10,746 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:10,746 DEBUG [Thread-3389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:58:10,746 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:10,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:10,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:10,746 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:10,746 DEBUG [Thread-3389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:58:10,747 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:10,847 DEBUG [Thread-3389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:58:10,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:58:10,848 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:58:10,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:58:10,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:58:10,849 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:58:10,849 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:10,849 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:58:10,849 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:58:10,850 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:10,850 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:10,850 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:10,850 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:10,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:10,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:10,850 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:10,850 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:58:10,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:10,851 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:10,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:58:10,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:10,851 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:58:10,851 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:58:10,851 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:58:10,851 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:10,851 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:58:10,851 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:58:10,851 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:58:10,851 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:10,852 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. started... 2023-03-21 11:58:10,852 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. as already flushing 2023-03-21 11:58:10,852 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:10,852 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:58:10,852 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:58:10,852 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:10,852 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:58:10,852 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:58:10,852 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:10,852 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:10,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:10,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:10,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:10,853 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:10,853 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:58:10,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:10,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:10,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:10,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:10,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:10,854 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:58:10,854 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@733a5f71[Count = 0] remaining members to acquire global barrier 2023-03-21 11:58:10,854 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:58:10,854 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:10,855 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:10,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:10,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:10,855 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:10,855 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:58:10,855 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:58:10,855 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:58:10,855 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:58:10,856 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:10,856 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:58:10,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:10,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:10,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:10,856 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:10,856 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:58:10,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:10,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:10,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:10,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:10,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:10,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:10,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:10,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:58:10,858 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:58:10,858 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:58:10,858 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:58:10,858 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:58:10,858 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:58:10,858 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:58:10,859 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:10,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:10,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:10,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:10,859 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:10,859 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:10,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:10,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:10,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:10,859 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:10,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:10,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:10,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:10,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:10,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:10,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:10,860 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:10,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:10,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:10,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:10,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:10,861 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:10,861 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:10,861 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:10,861 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:10,861 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:10,861 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:10,861 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:10,861 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:58:10,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:10,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:10,861 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:10,861 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:58:10,861 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:58:10,861 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:10,861 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:10,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:10,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:10,862 DEBUG [Thread-3389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:58:10,862 DEBUG [Thread-3389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:58:10,903 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:10,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55174 deadline: 1679399950902, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:10,904 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:10,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55130 deadline: 1679399950903, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:10,905 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:10,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55146 deadline: 1679399950904, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:10,905 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:10,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55178 deadline: 1679399950904, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:10,907 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:10,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55112 deadline: 1679399950904, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:10,962 DEBUG [Thread-3389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:58:10,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:58:10,964 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:58:10,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:58:10,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:58:10,965 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:58:10,965 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:10,965 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:58:10,965 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:58:10,965 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:10,965 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:10,965 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:10,965 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:10,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:10,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:10,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:10,966 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:10,966 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:58:10,966 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:10,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:58:10,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:10,966 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:58:10,966 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:58:10,966 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:58:10,966 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:58:10,966 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:10,967 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:58:10,967 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:58:10,967 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:58:10,967 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:10,967 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. started... 2023-03-21 11:58:10,967 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. as already flushing 2023-03-21 11:58:10,967 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:10,967 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:58:10,967 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:58:10,967 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:10,967 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:58:10,967 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:58:10,968 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:10,968 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:10,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:10,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:10,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:10,968 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:10,968 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:58:10,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:10,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:10,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:10,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:10,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:10,969 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:58:10,969 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@570d3898[Count = 0] remaining members to acquire global barrier 2023-03-21 11:58:10,969 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:58:10,969 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:10,969 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:10,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:10,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:10,969 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:10,969 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:58:10,969 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:58:10,969 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:58:10,969 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:58:10,970 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:58:10,970 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:10,970 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:10,970 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:58:10,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:10,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:10,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:10,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:10,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:10,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:10,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:10,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:10,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:10,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:10,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:58:10,972 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:58:10,972 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:58:10,972 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:58:10,972 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:58:10,972 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:58:10,972 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:58:10,972 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:10,972 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:10,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:10,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:10,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:10,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:10,972 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:10,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:10,973 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:10,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:10,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:10,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:10,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:10,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:10,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:10,974 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:10,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:10,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:10,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:10,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:10,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:10,975 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:10,975 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:10,975 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:10,975 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:10,975 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:10,975 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:10,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:10,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:10,975 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:58:10,975 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:58:10,975 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:10,975 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:10,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:10,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:10,975 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:10,975 DEBUG [Thread-3389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:58:10,976 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:10,976 DEBUG [Thread-3389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:58:11,020 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=51 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/C/937f89cb70ef470ab1405db22ca41972 2023-03-21 11:58:11,023 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/A/4240e15c12b04d3e81f24438d9a29734 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/4240e15c12b04d3e81f24438d9a29734 2023-03-21 11:58:11,026 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/4240e15c12b04d3e81f24438d9a29734, entries=200, sequenceid=51, filesize=31.9 K 2023-03-21 11:58:11,027 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/B/3d19e94189124e4cbc4e47a2465b3629 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/3d19e94189124e4cbc4e47a2465b3629 2023-03-21 11:58:11,036 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/3d19e94189124e4cbc4e47a2465b3629, entries=150, sequenceid=51, filesize=11.6 K 2023-03-21 11:58:11,037 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/C/937f89cb70ef470ab1405db22ca41972 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/937f89cb70ef470ab1405db22ca41972 2023-03-21 11:58:11,040 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/937f89cb70ef470ab1405db22ca41972, entries=150, sequenceid=51, filesize=11.6 K 2023-03-21 11:58:11,041 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=114.05 KB/116790 for cc711a8c4e0d80e60aabbf61e79ab961 in 459ms, sequenceid=51, compaction requested=true 2023-03-21 11:58:11,041 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for cc711a8c4e0d80e60aabbf61e79ab961: 2023-03-21 11:58:11,041 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-21 11:58:11,041 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-21 11:58:11,041 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-21 11:58:11,041 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-21 11:58:11,042 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-21 11:58:11,043 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 84178 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-21 11:58:11,043 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1912): cc711a8c4e0d80e60aabbf61e79ab961/A is initiating minor compaction (all files) 2023-03-21 11:58:11,043 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of cc711a8c4e0d80e60aabbf61e79ab961/A in TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:11,043 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/9bee3a786dea4a26bf8ed4cb00ead6a8, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/9842576d1c3f4d029f7e0d4bd1a2b1c1, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/4240e15c12b04d3e81f24438d9a29734] into tmpdir=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp, totalSize=82.2 K 2023-03-21 11:58:11,043 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-21 11:58:11,043 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1912): cc711a8c4e0d80e60aabbf61e79ab961/B is initiating minor compaction (all files) 2023-03-21 11:58:11,044 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegion(2259): Starting compaction of cc711a8c4e0d80e60aabbf61e79ab961/B in TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:11,044 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/ab7aacb9bff24b1aa02f840ba3d309ed, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/709cf3f129ee491ca2dce049878626f7, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/3d19e94189124e4cbc4e47a2465b3629] into tmpdir=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp, totalSize=34.8 K 2023-03-21 11:58:11,044 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting ab7aacb9bff24b1aa02f840ba3d309ed, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=17, earliestPutTs=1679399888333 2023-03-21 11:58:11,044 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 709cf3f129ee491ca2dce049878626f7, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=37, earliestPutTs=1679399888347 2023-03-21 11:58:11,045 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 3d19e94189124e4cbc4e47a2465b3629, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1679399890473 2023-03-21 11:58:11,049 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 9bee3a786dea4a26bf8ed4cb00ead6a8, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=17, earliestPutTs=1679399888333 2023-03-21 11:58:11,049 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 9842576d1c3f4d029f7e0d4bd1a2b1c1, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=37, earliestPutTs=1679399888347 2023-03-21 11:58:11,050 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 4240e15c12b04d3e81f24438d9a29734, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1679399890473 2023-03-21 11:58:11,052 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] throttle.PressureAwareThroughputController(145): cc711a8c4e0d80e60aabbf61e79ab961#B#compaction#441 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 57.69 MB/second 2023-03-21 11:58:11,071 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] throttle.PressureAwareThroughputController(145): cc711a8c4e0d80e60aabbf61e79ab961#A#compaction#442 average throughput is 1.16 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 57.69 MB/second 2023-03-21 11:58:11,076 DEBUG [Thread-3389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:58:11,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:58:11,077 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:58:11,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:58:11,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:58:11,082 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:58:11,082 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:11,082 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:58:11,082 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:58:11,082 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:11,082 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:11,083 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:11,083 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:11,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:11,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:11,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:11,084 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:11,084 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:11,084 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:58:11,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:58:11,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:11,084 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:58:11,087 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/B/62536a8df9444844907c95116eb74784 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/62536a8df9444844907c95116eb74784 2023-03-21 11:58:11,091 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in cc711a8c4e0d80e60aabbf61e79ab961/B of cc711a8c4e0d80e60aabbf61e79ab961 into 62536a8df9444844907c95116eb74784(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-21 11:58:11,092 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for cc711a8c4e0d80e60aabbf61e79ab961: 2023-03-21 11:58:11,092 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961., storeName=cc711a8c4e0d80e60aabbf61e79ab961/B, priority=13, startTime=1679399891041; duration=0sec 2023-03-21 11:58:11,092 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-21 11:58:11,092 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-21 11:58:11,093 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-21 11:58:11,093 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1912): cc711a8c4e0d80e60aabbf61e79ab961/C is initiating minor compaction (all files) 2023-03-21 11:58:11,093 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegion(2259): Starting compaction of cc711a8c4e0d80e60aabbf61e79ab961/C in TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:11,093 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:58:11,093 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/ae2c3046634b4a2dbf982036bddb03d9, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/fa5e691f8c3b4f18b9a12e51e5392da5, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/937f89cb70ef470ab1405db22ca41972] into tmpdir=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp, totalSize=34.8 K 2023-03-21 11:58:11,093 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:58:11,094 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:58:11,094 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:11,094 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:58:11,095 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting ae2c3046634b4a2dbf982036bddb03d9, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=17, earliestPutTs=1679399888333 2023-03-21 11:58:11,095 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:58:11,095 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:11,095 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:58:11,095 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. started... 2023-03-21 11:58:11,095 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting fa5e691f8c3b4f18b9a12e51e5392da5, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=37, earliestPutTs=1679399888347 2023-03-21 11:58:11,095 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing cc711a8c4e0d80e60aabbf61e79ab961 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-21 11:58:11,095 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cc711a8c4e0d80e60aabbf61e79ab961, store=A 2023-03-21 11:58:11,095 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:58:11,095 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 937f89cb70ef470ab1405db22ca41972, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1679399890473 2023-03-21 11:58:11,095 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cc711a8c4e0d80e60aabbf61e79ab961, store=B 2023-03-21 11:58:11,095 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:58:11,095 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cc711a8c4e0d80e60aabbf61e79ab961, store=C 2023-03-21 11:58:11,095 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:58:11,100 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] throttle.PressureAwareThroughputController(145): cc711a8c4e0d80e60aabbf61e79ab961#C#compaction#443 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 57.69 MB/second 2023-03-21 11:58:11,142 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/C/2491db6eee204fdd84376f13fb75c149 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/2491db6eee204fdd84376f13fb75c149 2023-03-21 11:58:11,144 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230321329205302a41469496283ad325c1d341 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230321329205302a41469496283ad325c1d341 2023-03-21 11:58:11,149 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in cc711a8c4e0d80e60aabbf61e79ab961/C of cc711a8c4e0d80e60aabbf61e79ab961 into 2491db6eee204fdd84376f13fb75c149(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-21 11:58:11,149 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for cc711a8c4e0d80e60aabbf61e79ab961: 2023-03-21 11:58:11,149 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961., storeName=cc711a8c4e0d80e60aabbf61e79ab961/C, priority=13, startTime=1679399891042; duration=0sec 2023-03-21 11:58:11,149 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-21 11:58:11,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(9158): Flush requested on cc711a8c4e0d80e60aabbf61e79ab961 2023-03-21 11:58:11,206 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. as already flushing 2023-03-21 11:58:11,214 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:11,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55146 deadline: 1679399951211, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:11,215 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:11,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55178 deadline: 1679399951211, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:11,215 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:11,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55174 deadline: 1679399951212, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:11,215 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:11,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55112 deadline: 1679399951213, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:11,215 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:11,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55130 deadline: 1679399951213, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:11,315 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:11,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55146 deadline: 1679399951315, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:11,316 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:11,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55178 deadline: 1679399951315, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:11,316 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:11,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55174 deadline: 1679399951316, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:11,318 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:11,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55112 deadline: 1679399951316, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:11,318 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:11,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55130 deadline: 1679399951316, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:11,518 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:11,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55174 deadline: 1679399951517, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:11,519 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:11,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55146 deadline: 1679399951517, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:11,519 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:11,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55178 deadline: 1679399951517, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:11,520 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:11,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55130 deadline: 1679399951519, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:11,521 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:11,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55112 deadline: 1679399951519, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:11,522 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/A/92d9338ebb4740c481ddd1dcf985d71d as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/92d9338ebb4740c481ddd1dcf985d71d 2023-03-21 11:58:11,525 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in cc711a8c4e0d80e60aabbf61e79ab961/A of cc711a8c4e0d80e60aabbf61e79ab961 into 92d9338ebb4740c481ddd1dcf985d71d(size=25.2 K), total size for store is 25.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-21 11:58:11,525 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for cc711a8c4e0d80e60aabbf61e79ab961: 2023-03-21 11:58:11,525 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961., storeName=cc711a8c4e0d80e60aabbf61e79ab961/A, priority=13, startTime=1679399891041; duration=0sec 2023-03-21 11:58:11,525 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-21 11:58:11,551 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=72, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/A/fe27dff1472d4ce59d37bb5d641342f8 2023-03-21 11:58:11,821 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:11,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55178 deadline: 1679399951820, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:11,822 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:11,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55174 deadline: 1679399951821, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:11,823 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:11,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55146 deadline: 1679399951821, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:11,823 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:11,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55130 deadline: 1679399951822, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:11,824 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:11,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55112 deadline: 1679399951823, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:11,959 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=72 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/B/62399949fee641a395f900f5c357b5d9 2023-03-21 11:58:12,323 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:12,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55178 deadline: 1679399952322, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:12,324 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:12,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55130 deadline: 1679399952324, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:12,326 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:12,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55174 deadline: 1679399952325, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:12,329 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:12,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55146 deadline: 1679399952327, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:12,330 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:12,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55112 deadline: 1679399952329, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:12,369 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=72 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/C/4bcb4e67df8c4d97b62f9c129502c526 2023-03-21 11:58:12,372 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/A/fe27dff1472d4ce59d37bb5d641342f8 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/fe27dff1472d4ce59d37bb5d641342f8 2023-03-21 11:58:12,374 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/fe27dff1472d4ce59d37bb5d641342f8, entries=150, sequenceid=72, filesize=25.1 K 2023-03-21 11:58:12,375 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/B/62399949fee641a395f900f5c357b5d9 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/62399949fee641a395f900f5c357b5d9 2023-03-21 11:58:12,380 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/62399949fee641a395f900f5c357b5d9, entries=150, sequenceid=72, filesize=11.6 K 2023-03-21 11:58:12,381 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/C/4bcb4e67df8c4d97b62f9c129502c526 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/4bcb4e67df8c4d97b62f9c129502c526 2023-03-21 11:58:12,389 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/4bcb4e67df8c4d97b62f9c129502c526, entries=150, sequenceid=72, filesize=11.6 K 2023-03-21 11:58:12,389 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for cc711a8c4e0d80e60aabbf61e79ab961 in 1294ms, sequenceid=72, compaction requested=false 2023-03-21 11:58:12,390 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for cc711a8c4e0d80e60aabbf61e79ab961: 2023-03-21 11:58:12,390 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:12,390 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:58:12,390 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:58:12,390 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:12,390 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:58:12,390 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:58:12,391 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:12,391 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:12,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:12,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:12,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:12,391 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:12,391 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:58:12,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:12,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:12,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:12,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:12,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:12,392 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:58:12,392 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64b09f4f[Count = 0] remaining members to acquire global barrier 2023-03-21 11:58:12,392 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:58:12,392 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:12,393 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:12,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:12,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:12,393 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:58:12,393 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:58:12,393 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:12,393 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:58:12,393 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:58:12,394 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:12,394 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:58:12,394 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:12,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:12,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:12,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:12,394 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:58:12,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:12,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:12,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:12,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:12,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:12,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:12,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:12,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:58:12,396 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:58:12,396 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:58:12,396 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:58:12,396 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:58:12,396 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:58:12,396 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:58:12,397 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:12,397 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:12,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:12,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:12,397 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:12,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:12,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:12,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:12,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:12,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:12,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:12,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:12,398 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:12,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:12,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:12,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:12,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:12,399 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:12,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:12,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:12,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:12,400 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:12,400 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:12,400 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:12,401 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:12,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:12,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:12,401 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:12,401 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:12,401 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:12,401 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:58:12,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:12,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:12,401 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:12,401 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:58:12,401 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:12,401 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:12,401 DEBUG [Thread-3389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:58:12,401 DEBUG [Thread-3389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:58:12,501 DEBUG [Thread-3389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:58:12,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:58:12,503 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:58:12,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:58:12,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:58:12,504 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:58:12,504 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:12,504 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:58:12,504 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:58:12,504 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:12,504 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:12,505 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:12,505 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:12,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:12,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:12,505 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:12,505 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:58:12,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:12,505 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:12,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:58:12,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:12,506 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:58:12,506 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:58:12,506 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:58:12,506 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:58:12,506 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:12,506 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:58:12,506 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:58:12,506 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:58:12,506 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:12,506 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. started... 2023-03-21 11:58:12,506 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing cc711a8c4e0d80e60aabbf61e79ab961 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-21 11:58:12,506 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cc711a8c4e0d80e60aabbf61e79ab961, store=A 2023-03-21 11:58:12,507 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:58:12,507 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cc711a8c4e0d80e60aabbf61e79ab961, store=B 2023-03-21 11:58:12,507 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:58:12,507 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cc711a8c4e0d80e60aabbf61e79ab961, store=C 2023-03-21 11:58:12,507 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:58:12,524 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230321fd4ed93a970242feac034fbeaaae2452 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230321fd4ed93a970242feac034fbeaaae2452 2023-03-21 11:58:12,528 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=86, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/A/c486aa6e3be947f9aa1c212c84045f2a 2023-03-21 11:58:12,536 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=86 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/B/f586393140554ad19d819ac9da0c00fe 2023-03-21 11:58:12,952 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=86 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/C/6045696af9744b4f91b718d1213352fc 2023-03-21 11:58:12,955 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/A/c486aa6e3be947f9aa1c212c84045f2a as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/c486aa6e3be947f9aa1c212c84045f2a 2023-03-21 11:58:12,957 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/c486aa6e3be947f9aa1c212c84045f2a, entries=150, sequenceid=86, filesize=25.1 K 2023-03-21 11:58:12,958 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/B/f586393140554ad19d819ac9da0c00fe as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/f586393140554ad19d819ac9da0c00fe 2023-03-21 11:58:12,963 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/f586393140554ad19d819ac9da0c00fe, entries=150, sequenceid=86, filesize=11.6 K 2023-03-21 11:58:12,966 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/C/6045696af9744b4f91b718d1213352fc as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/6045696af9744b4f91b718d1213352fc 2023-03-21 11:58:12,983 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/6045696af9744b4f91b718d1213352fc, entries=150, sequenceid=86, filesize=11.6 K 2023-03-21 11:58:12,984 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=0 B/0 for cc711a8c4e0d80e60aabbf61e79ab961 in 478ms, sequenceid=86, compaction requested=true 2023-03-21 11:58:12,984 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for cc711a8c4e0d80e60aabbf61e79ab961: 2023-03-21 11:58:12,984 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:12,984 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:58:12,984 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:58:12,984 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:12,984 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:58:12,984 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:58:12,985 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:12,985 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:12,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:12,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:12,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:12,985 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:12,985 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:58:12,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:12,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:12,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:12,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:12,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:12,987 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:58:12,987 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3606f0da[Count = 0] remaining members to acquire global barrier 2023-03-21 11:58:12,987 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:58:12,987 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:12,988 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:12,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:12,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:12,988 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:58:12,988 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:58:12,988 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:58:12,988 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:12,988 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:58:12,989 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:12,989 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:58:12,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:12,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:12,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:12,989 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:12,990 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:58:12,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:12,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:12,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:12,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:12,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:12,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:12,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:12,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:58:12,992 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:58:12,992 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:58:12,992 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:58:12,992 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:58:12,992 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:58:12,992 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:58:12,993 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:12,993 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:12,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:12,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:12,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:12,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:12,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:12,993 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:12,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:12,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:12,993 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:12,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:12,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:12,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:12,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:12,997 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:12,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:12,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:12,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:12,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:12,999 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:12,999 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:13,000 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:58:13,000 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:58:13,000 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:13,000 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:13,000 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:13,000 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,000 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:58:13,000 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:13,000 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:13,000 DEBUG [Thread-3389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:58:13,000 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:13,000 DEBUG [Thread-3389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:58:13,000 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:13,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:13,002 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:58:13,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:13,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:13,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:13,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:13,100 DEBUG [Thread-3389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:58:13,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:58:13,102 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:58:13,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:58:13,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:58:13,110 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:58:13,110 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:13,110 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:58:13,110 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:58:13,110 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:13,110 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:13,111 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,111 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:13,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:13,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:13,111 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,111 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:58:13,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:13,112 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:13,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:58:13,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:13,112 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:58:13,118 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:58:13,118 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:58:13,118 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:58:13,118 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:13,118 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:58:13,118 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:58:13,118 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:58:13,118 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:13,118 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. started... 2023-03-21 11:58:13,118 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for cc711a8c4e0d80e60aabbf61e79ab961: 2023-03-21 11:58:13,119 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:13,119 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:58:13,119 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:58:13,119 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,119 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:58:13,119 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:58:13,123 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,123 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:13,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:13,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:13,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:13,124 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:13,124 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:58:13,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:13,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:13,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:13,125 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:58:13,125 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1766e0bc[Count = 0] remaining members to acquire global barrier 2023-03-21 11:58:13,125 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:58:13,125 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:13,126 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:13,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:13,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:13,126 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:58:13,126 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:58:13,126 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:58:13,126 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,126 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:58:13,127 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,127 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:58:13,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:13,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:13,127 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:13,127 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:58:13,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:13,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:13,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:13,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:13,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:13,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:58:13,129 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:58:13,129 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:58:13,129 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:58:13,129 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:58:13,129 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:58:13,129 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:58:13,130 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:13,130 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:13,130 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:13,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:13,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:13,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:13,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:13,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:13,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:13,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:13,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:13,131 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:13,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:13,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:13,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:13,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,132 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:13,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:13,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,134 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,134 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:13,134 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:13,134 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:13,134 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:58:13,134 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:58:13,134 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,134 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:13,134 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:13,134 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:13,134 DEBUG [Thread-3389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:58:13,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:13,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:13,134 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:13,134 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:13,134 DEBUG [Thread-3389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:58:13,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:13,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:13,235 DEBUG [Thread-3389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:58:13,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:58:13,236 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:58:13,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:58:13,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:58:13,247 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:58:13,247 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:13,247 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:58:13,247 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:58:13,248 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:13,248 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:13,248 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,249 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,249 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:58:13,249 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:13,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:13,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:13,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:13,250 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:13,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:58:13,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:13,250 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:58:13,250 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:58:13,250 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:58:13,250 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:13,250 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:58:13,251 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:58:13,251 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:58:13,251 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:13,251 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. started... 2023-03-21 11:58:13,251 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for cc711a8c4e0d80e60aabbf61e79ab961: 2023-03-21 11:58:13,251 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:13,251 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:58:13,251 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:58:13,251 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,251 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:58:13,251 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:58:13,252 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:13,252 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:13,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:13,253 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:13,253 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:58:13,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:13,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:13,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:13,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:13,254 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:58:13,254 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2991c330[Count = 0] remaining members to acquire global barrier 2023-03-21 11:58:13,254 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:58:13,254 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:13,255 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:13,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:13,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:13,255 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:58:13,255 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:58:13,255 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:58:13,255 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,255 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:58:13,256 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,256 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:58:13,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:13,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:13,256 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:13,256 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:58:13,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:13,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:13,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:13,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:13,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:13,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:58:13,258 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:58:13,258 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:58:13,258 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:58:13,258 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:58:13,258 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:58:13,258 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:58:13,259 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:13,259 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:13,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:13,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:13,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:13,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:13,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:13,259 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:13,259 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:13,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:13,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:13,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:13,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:13,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:13,260 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:13,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:13,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:13,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,262 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:13,262 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:13,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:13,262 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:13,262 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:13,262 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:13,262 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:58:13,262 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:58:13,262 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:13,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:13,262 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:58:13,262 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,262 DEBUG [Thread-3389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:58:13,262 DEBUG [Thread-3389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:58:13,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:13,262 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:13,263 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:13,263 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:13,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(9158): Flush requested on cc711a8c4e0d80e60aabbf61e79ab961 2023-03-21 11:58:13,354 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing cc711a8c4e0d80e60aabbf61e79ab961 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-21 11:58:13,354 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cc711a8c4e0d80e60aabbf61e79ab961, store=A 2023-03-21 11:58:13,354 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:58:13,354 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cc711a8c4e0d80e60aabbf61e79ab961, store=B 2023-03-21 11:58:13,355 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:58:13,355 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cc711a8c4e0d80e60aabbf61e79ab961, store=C 2023-03-21 11:58:13,355 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:58:13,363 DEBUG [Thread-3389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:58:13,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:58:13,364 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:58:13,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:58:13,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:58:13,365 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:58:13,365 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:13,365 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:58:13,365 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:58:13,365 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:13,366 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:13,366 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,366 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:13,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:13,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:13,366 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,366 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:58:13,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:13,367 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:13,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:58:13,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:13,367 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:58:13,367 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:58:13,367 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:58:13,367 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:58:13,367 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:13,367 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:58:13,367 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:58:13,367 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:58:13,367 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:13,367 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. started... 2023-03-21 11:58:13,368 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. as already flushing 2023-03-21 11:58:13,368 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:13,368 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:58:13,368 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:58:13,368 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,368 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:58:13,368 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:58:13,369 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:13,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:13,369 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:13,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:13,369 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:13,369 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:58:13,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:13,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:13,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:13,370 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:58:13,371 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@493b7baa[Count = 0] remaining members to acquire global barrier 2023-03-21 11:58:13,371 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:58:13,371 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:13,371 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,372 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:58:13,372 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:13,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:13,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:13,372 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:58:13,372 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:58:13,372 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:58:13,373 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:58:13,373 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:13,373 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:58:13,373 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:13,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:13,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:13,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:13,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:13,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:13,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:13,375 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:13,375 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:13,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55146 deadline: 1679399953370, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55130 deadline: 1679399953370, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,375 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679399893368, max timestamp=1679399893374Num uniques 859; 2023-03-21 11:58:13,375 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-21 11:58:13,375 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679399893368, max timestamp=1679399893374Num uniques 859; 2023-03-21 11:58:13,375 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-21 11:58:13,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,376 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679399893368, max timestamp=1679399893374Num uniques 859; 2023-03-21 11:58:13,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:58:13,376 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-21 11:58:13,376 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:58:13,376 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:58:13,376 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:58:13,376 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:58:13,376 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:58:13,376 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:58:13,377 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:13,377 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:13,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:13,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:13,377 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:13,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:13,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:13,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:13,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:13,377 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:13,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:13,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:13,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:13,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:13,378 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:13,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:13,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:13,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,380 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,380 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:13,380 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:13,380 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:13,381 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:13,381 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:58:13,381 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:58:13,381 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:13,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:13,381 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:13,381 DEBUG [Thread-3389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:58:13,381 DEBUG [Thread-3389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:58:13,381 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:58:13,381 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:13,381 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:13,381 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:13,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:13,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:13,393 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:13,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55112 deadline: 1679399953391, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,394 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:13,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55174 deadline: 1679399953392, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,395 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:13,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55178 deadline: 1679399953393, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,478 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:13,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55146 deadline: 1679399953476, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,478 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:13,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55130 deadline: 1679399953476, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,481 DEBUG [Thread-3389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:58:13,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:58:13,483 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:58:13,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:58:13,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:58:13,484 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:58:13,484 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:13,484 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:58:13,484 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:58:13,484 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:13,484 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:13,485 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:13,485 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:13,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:13,485 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,485 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:58:13,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:13,485 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:13,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:58:13,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:13,485 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:58:13,486 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:58:13,486 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:58:13,486 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:58:13,486 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:13,486 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:58:13,486 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:58:13,486 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:58:13,486 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:13,486 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. started... 2023-03-21 11:58:13,486 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. as already flushing 2023-03-21 11:58:13,486 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:13,486 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:58:13,486 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:58:13,486 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,486 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:58:13,486 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:58:13,487 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,487 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:13,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:13,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:13,487 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:13,487 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:58:13,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:13,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:13,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:13,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:13,488 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:58:13,488 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@48bad277[Count = 0] remaining members to acquire global barrier 2023-03-21 11:58:13,488 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:58:13,488 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:13,489 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:13,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:13,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:13,489 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:58:13,489 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:58:13,489 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,489 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:58:13,489 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:58:13,490 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,490 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:58:13,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:13,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:13,490 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:13,490 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:58:13,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:13,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:13,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:13,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:13,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:13,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:58:13,492 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:58:13,492 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:58:13,492 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:58:13,492 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:58:13,492 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:58:13,492 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:58:13,493 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:13,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:13,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:13,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:13,493 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:13,493 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:13,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:13,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:13,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:13,493 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:13,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:13,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:13,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:13,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:13,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:13,494 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:13,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:13,495 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:13,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55112 deadline: 1679399953494, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,496 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,496 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:13,496 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:13,496 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:58:13,496 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:58:13,496 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:58:13,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55174 deadline: 1679399953494, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,496 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:13,496 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:13,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:13,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:13,496 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:13,496 DEBUG [Thread-3389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:58:13,496 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:13,496 DEBUG [Thread-3389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:58:13,497 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,497 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:13,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:13,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:13,497 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:13,497 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:13,498 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:13,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55178 deadline: 1679399953496, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,597 DEBUG [Thread-3389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:58:13,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:58:13,598 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:58:13,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:58:13,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:58:13,599 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:58:13,599 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:13,599 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:58:13,599 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:58:13,600 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:13,600 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:13,600 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:13,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:13,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:13,600 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:13,600 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,600 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:58:13,601 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:13,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:58:13,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:13,601 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:58:13,601 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:58:13,601 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:58:13,601 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:58:13,601 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:13,601 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:58:13,601 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:58:13,601 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:58:13,601 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:13,601 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. started... 2023-03-21 11:58:13,602 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. as already flushing 2023-03-21 11:58:13,602 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:13,602 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:58:13,602 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:58:13,602 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,602 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:58:13,602 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:58:13,603 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,603 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:13,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:13,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:13,603 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:13,603 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:58:13,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:13,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:13,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:13,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:13,604 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:58:13,604 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6879b8dc[Count = 0] remaining members to acquire global barrier 2023-03-21 11:58:13,604 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:58:13,604 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:13,604 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:13,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:13,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:13,604 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,604 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:58:13,604 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:58:13,605 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:58:13,605 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:58:13,605 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,605 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:58:13,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:13,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:13,605 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:13,605 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:58:13,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:13,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:13,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:13,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:13,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:13,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:58:13,607 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:58:13,607 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:58:13,607 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:58:13,607 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:58:13,607 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:58:13,607 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:58:13,620 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:13,620 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:13,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:13,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:13,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:13,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:13,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:13,620 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:13,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:13,621 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:13,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:13,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:13,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:13,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:13,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:13,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,622 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:13,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:13,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,623 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,623 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:13,623 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:13,623 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:58:13,623 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:58:13,623 DEBUG [Thread-3389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:58:13,623 DEBUG [Thread-3389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:58:13,623 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:13,623 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,623 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:13,623 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:13,623 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:13,624 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:13,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:13,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:13,623 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:13,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:13,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:13,680 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:13,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55146 deadline: 1679399953679, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,681 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:13,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55130 deadline: 1679399953679, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,698 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:13,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55112 deadline: 1679399953697, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,700 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:13,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55174 deadline: 1679399953698, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,701 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:13,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55178 deadline: 1679399953699, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,723 DEBUG [Thread-3389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:58:13,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:58:13,725 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:58:13,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:58:13,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:58:13,726 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:58:13,726 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:13,726 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:58:13,726 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:58:13,726 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:13,726 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:13,727 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,727 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:13,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:13,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:13,727 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,727 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:58:13,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:13,727 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:13,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:58:13,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:13,728 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:58:13,728 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:58:13,728 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:58:13,728 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:58:13,728 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:13,728 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:58:13,728 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:58:13,728 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:58:13,728 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:13,729 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. started... 2023-03-21 11:58:13,729 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. as already flushing 2023-03-21 11:58:13,729 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:13,729 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:58:13,729 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:58:13,729 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,729 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:58:13,729 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:58:13,729 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,729 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:13,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:13,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:13,730 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:13,730 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:58:13,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:13,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:13,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:13,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:13,731 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:58:13,731 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:58:13,731 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4fd66ea0[Count = 0] remaining members to acquire global barrier 2023-03-21 11:58:13,731 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:13,731 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:13,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:13,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:13,731 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:58:13,731 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:58:13,731 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:58:13,731 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,732 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:58:13,732 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,732 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:58:13,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:13,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:13,732 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:13,732 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:58:13,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:13,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:13,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:13,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:13,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:13,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:58:13,734 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:58:13,734 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:58:13,734 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:58:13,734 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:58:13,734 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:58:13,734 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:58:13,734 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:13,735 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:13,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:13,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:13,735 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:13,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:13,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:13,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:13,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:13,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:13,735 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:13,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:13,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:13,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:13,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:13,736 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:13,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:13,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,738 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:13,738 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:13,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:13,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:13,738 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:13,738 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:58:13,738 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:58:13,738 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,738 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:58:13,738 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:13,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:13,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:13,738 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:13,738 DEBUG [Thread-3389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:58:13,738 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,738 DEBUG [Thread-3389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:58:13,738 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:13,738 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:13,738 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:13,764 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303214d0f304aff7d4dd2858bec6fcaba27d5 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303214d0f304aff7d4dd2858bec6fcaba27d5 2023-03-21 11:58:13,838 DEBUG [Thread-3389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:58:13,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:58:13,840 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:58:13,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:58:13,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:58:13,841 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:58:13,841 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:13,841 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:58:13,841 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:58:13,841 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:13,841 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:13,842 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,842 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:13,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:13,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:13,842 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,842 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:58:13,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:13,843 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:13,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:58:13,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:13,843 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:58:13,843 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:58:13,843 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:58:13,843 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:58:13,843 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:13,843 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:58:13,843 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:58:13,843 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:58:13,843 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:13,843 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. started... 2023-03-21 11:58:13,843 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. as already flushing 2023-03-21 11:58:13,843 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:13,844 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:58:13,844 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:58:13,844 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,844 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:58:13,844 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:58:13,844 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,844 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:13,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:13,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:13,845 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:13,845 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:58:13,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:13,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:13,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:13,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:13,846 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:58:13,846 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5da2badb[Count = 0] remaining members to acquire global barrier 2023-03-21 11:58:13,846 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:58:13,846 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:13,846 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:13,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:13,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:13,846 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:58:13,846 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:58:13,846 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:58:13,846 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,846 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:58:13,847 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:13,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:13,847 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:58:13,847 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:13,847 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:58:13,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:13,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:13,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:13,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:13,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:13,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:58:13,849 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:58:13,849 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:58:13,849 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:58:13,849 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:58:13,849 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:58:13,849 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:58:13,850 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:13,850 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:13,850 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:13,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:13,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:13,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:13,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:13,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:13,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:13,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:13,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:13,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:13,851 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:13,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:13,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:13,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,851 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:13,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:13,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,853 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,853 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:13,853 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:13,853 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,853 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:13,853 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:13,853 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:13,853 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:13,853 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:58:13,853 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:58:13,853 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:13,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:13,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:13,853 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:13,854 DEBUG [Thread-3389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:58:13,854 DEBUG [Thread-3389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:58:13,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:13,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:13,954 DEBUG [Thread-3389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:58:13,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:58:13,955 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:58:13,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:58:13,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:58:13,956 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:58:13,956 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:13,956 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:58:13,956 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:58:13,957 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:13,957 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:13,957 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:13,957 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:13,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:13,958 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:13,958 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:58:13,958 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:13,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:58:13,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:13,958 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:58:13,958 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:58:13,958 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:58:13,958 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:58:13,958 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:13,958 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:58:13,959 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:58:13,959 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:58:13,959 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:13,959 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. started... 2023-03-21 11:58:13,959 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. as already flushing 2023-03-21 11:58:13,959 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:13,959 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:58:13,959 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:58:13,959 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,959 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:58:13,959 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:58:13,959 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:13,959 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:13,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:13,960 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:13,960 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:58:13,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:13,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:13,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:13,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:13,961 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:58:13,961 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4315f272[Count = 0] remaining members to acquire global barrier 2023-03-21 11:58:13,961 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:58:13,961 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:13,961 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:13,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:13,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:13,961 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:58:13,961 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,961 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:58:13,961 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:58:13,961 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:58:13,962 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:58:13,962 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:13,962 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,962 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:58:13,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:13,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:13,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:13,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:13,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:13,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:13,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:13,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:58:13,964 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:58:13,964 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:58:13,964 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:58:13,964 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:58:13,964 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:58:13,964 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:58:13,964 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:13,964 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:13,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:13,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:13,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:13,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:13,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:13,964 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:13,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:13,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:13,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:13,965 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:13,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:13,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:13,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:13,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,966 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:13,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:13,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,967 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,967 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:13,967 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:13,967 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:13,967 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:58:13,967 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:58:13,967 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:13,967 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,967 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:13,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:13,967 DEBUG [Thread-3389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:58:13,967 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:13,967 DEBUG [Thread-3389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:58:13,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:13,967 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:13,967 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:13,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:13,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:13,982 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:13,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55146 deadline: 1679399953981, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:13,986 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:13,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55130 deadline: 1679399953982, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,000 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:14,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55112 deadline: 1679399954000, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,004 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:14,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55178 deadline: 1679399954002, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,005 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:14,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55174 deadline: 1679399954003, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,067 DEBUG [Thread-3389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:58:14,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:58:14,069 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:58:14,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:58:14,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:58:14,070 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:58:14,070 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:14,070 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:58:14,070 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:58:14,070 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:14,071 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:14,071 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,071 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:14,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:14,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:14,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:14,072 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,072 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:58:14,072 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:14,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:58:14,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:14,072 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:58:14,072 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:58:14,072 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:58:14,072 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:58:14,072 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:14,072 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:58:14,072 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:58:14,072 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:58:14,072 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:14,073 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. started... 2023-03-21 11:58:14,073 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. as already flushing 2023-03-21 11:58:14,073 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:14,073 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:58:14,073 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:58:14,073 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,073 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:58:14,073 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:58:14,074 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,074 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:14,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:14,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:14,074 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:14,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:14,074 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:58:14,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:14,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:14,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:14,075 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:58:14,075 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@668f0d94[Count = 0] remaining members to acquire global barrier 2023-03-21 11:58:14,075 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:58:14,075 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:14,075 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:14,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:14,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:14,076 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:58:14,076 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:58:14,076 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:58:14,076 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,076 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:58:14,076 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:58:14,076 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:14,076 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:14,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:14,076 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:58:14,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:14,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:14,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:14,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:14,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:14,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:58:14,078 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:58:14,078 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:58:14,078 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:58:14,078 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:58:14,078 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:58:14,078 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:58:14,079 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:14,079 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:14,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:14,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:14,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:14,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:14,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:14,079 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:14,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:14,079 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:14,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:14,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:14,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:14,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:14,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:14,080 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:14,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:14,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,081 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,081 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:14,081 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:14,081 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:14,082 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:14,082 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:58:14,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:14,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:14,082 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:58:14,082 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:58:14,082 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,082 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:14,081 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:14,082 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:14,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:14,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:14,082 DEBUG [Thread-3389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:58:14,082 DEBUG [Thread-3389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:58:14,082 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:14,168 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=99, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/A/41db545ed1c544c88c15c6ce47280df2 2023-03-21 11:58:14,182 DEBUG [Thread-3389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:58:14,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:58:14,184 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:58:14,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:58:14,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:58:14,185 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:58:14,185 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:14,185 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:58:14,185 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:58:14,185 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:14,185 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:14,187 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,187 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:14,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:14,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:14,187 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,187 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:58:14,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:14,187 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:14,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:58:14,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:14,187 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:58:14,188 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:58:14,188 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:58:14,188 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:58:14,188 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:14,188 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:58:14,188 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:58:14,188 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:58:14,188 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:14,188 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. started... 2023-03-21 11:58:14,188 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. as already flushing 2023-03-21 11:58:14,188 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:14,188 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:58:14,188 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:58:14,188 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,188 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:58:14,188 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:58:14,189 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,189 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:14,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:14,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:14,189 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:14,189 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:58:14,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:14,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:14,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:14,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:14,190 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:58:14,190 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1a50dc8b[Count = 0] remaining members to acquire global barrier 2023-03-21 11:58:14,190 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:58:14,190 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:14,191 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:14,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:14,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:14,191 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,191 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:58:14,191 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:58:14,191 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:58:14,191 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:58:14,193 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:58:14,193 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:14,193 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,193 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:58:14,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:14,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:14,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:14,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:14,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:14,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:14,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:14,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:58:14,195 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:58:14,195 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:58:14,195 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:58:14,195 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:58:14,195 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:58:14,195 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:58:14,195 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:14,195 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:14,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:14,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:14,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:14,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:14,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:14,195 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:14,196 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:14,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:14,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:14,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:14,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:14,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:14,197 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:14,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:14,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:14,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,200 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,200 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:14,200 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:14,200 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:14,200 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:14,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:14,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:14,200 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,200 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:14,200 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:14,200 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:14,200 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:14,200 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:58:14,200 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:58:14,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:14,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:14,200 DEBUG [Thread-3389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:58:14,200 DEBUG [Thread-3389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:58:14,200 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:58:14,300 DEBUG [Thread-3389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:58:14,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:58:14,302 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:58:14,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:58:14,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:58:14,304 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:58:14,304 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:14,304 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:58:14,304 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:58:14,304 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:14,304 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:14,305 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:14,305 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:14,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:14,305 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,305 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:58:14,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:14,305 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:14,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:58:14,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:14,306 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:58:14,306 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:58:14,306 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:58:14,306 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:58:14,306 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:14,306 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:58:14,306 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:58:14,306 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:58:14,306 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:14,306 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. started... 2023-03-21 11:58:14,306 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. as already flushing 2023-03-21 11:58:14,306 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:14,306 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:58:14,307 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:58:14,307 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,307 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:58:14,307 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:58:14,307 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,307 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:14,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:14,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:14,308 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:14,308 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:58:14,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:14,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:14,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:14,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:14,309 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:58:14,309 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@526cd978[Count = 0] remaining members to acquire global barrier 2023-03-21 11:58:14,309 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:58:14,309 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:14,310 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:14,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:14,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:14,310 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:58:14,310 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,310 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:58:14,310 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:58:14,310 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:58:14,311 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:58:14,311 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,311 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:14,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:14,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:14,311 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:58:14,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:14,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:14,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:14,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:14,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:14,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:58:14,314 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:58:14,314 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:58:14,314 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:58:14,314 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:58:14,314 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:58:14,314 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:58:14,315 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:14,315 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:14,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:14,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:14,315 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:14,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:14,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:14,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:14,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:14,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:14,315 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:14,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:14,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:14,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:14,317 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:14,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:14,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:14,318 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:14,318 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:14,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:14,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:14,318 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,318 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:14,318 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:14,318 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:14,318 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:58:14,318 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:58:14,318 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,318 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:14,318 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:14,318 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:14,319 DEBUG [Thread-3389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:58:14,319 DEBUG [Thread-3389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:58:14,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:14,319 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:58:14,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:14,419 DEBUG [Thread-3389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:58:14,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:58:14,420 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:58:14,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:58:14,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:58:14,421 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:58:14,421 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:14,421 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:58:14,421 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:58:14,422 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:14,422 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:14,422 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:14,422 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:14,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:14,423 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,423 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:58:14,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:14,423 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:14,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:58:14,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:14,423 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:58:14,423 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:58:14,423 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:58:14,424 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:58:14,424 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:14,424 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:58:14,424 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:58:14,424 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:58:14,424 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:14,424 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. started... 2023-03-21 11:58:14,424 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. as already flushing 2023-03-21 11:58:14,424 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:14,424 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:58:14,424 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:58:14,424 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,424 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:58:14,424 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:58:14,425 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,425 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:14,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:14,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:14,425 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:14,425 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:58:14,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:14,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:14,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:14,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:14,426 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:58:14,426 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e862725[Count = 0] remaining members to acquire global barrier 2023-03-21 11:58:14,426 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:58:14,426 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:14,427 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:14,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:14,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:14,427 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:58:14,427 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:58:14,427 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:58:14,427 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,427 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:58:14,428 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,428 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:58:14,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:14,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:14,428 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:14,428 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:58:14,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:14,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:14,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:14,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:14,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:14,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:58:14,430 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:58:14,430 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:58:14,430 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:58:14,430 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:58:14,430 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:58:14,430 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:58:14,430 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:14,430 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:14,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:14,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:14,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:14,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:14,431 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:14,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:14,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:14,431 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:14,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:14,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:14,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:14,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:14,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:14,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,432 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:14,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:14,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,434 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,434 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:14,434 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:14,434 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:58:14,434 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:58:14,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:14,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:14,434 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:14,434 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:14,434 DEBUG [Thread-3389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:58:14,434 DEBUG [Thread-3389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:58:14,434 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:14,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:14,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:14,434 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,434 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:14,434 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:14,434 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:14,488 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:14,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55146 deadline: 1679399954487, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,492 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:14,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55130 deadline: 1679399954490, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,505 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:14,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55112 deadline: 1679399954501, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,506 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:14,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55178 deadline: 1679399954505, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,510 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:14,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55174 deadline: 1679399954508, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,534 DEBUG [Thread-3389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:58:14,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:58:14,536 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:58:14,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:58:14,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:58:14,537 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:58:14,537 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:14,537 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:58:14,537 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:58:14,537 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:14,537 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:14,538 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,538 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:14,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:14,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:14,538 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,538 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:58:14,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:14,538 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:14,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:58:14,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:14,539 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:58:14,539 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:58:14,539 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:58:14,539 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:58:14,539 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:14,539 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:58:14,539 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:58:14,539 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:58:14,539 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:14,539 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. started... 2023-03-21 11:58:14,539 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. as already flushing 2023-03-21 11:58:14,539 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:14,539 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:58:14,539 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:58:14,539 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,539 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:58:14,539 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:58:14,540 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:14,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:14,541 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:14,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:14,541 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:14,541 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:58:14,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:14,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:14,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:14,542 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:58:14,542 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@20fb7ecc[Count = 0] remaining members to acquire global barrier 2023-03-21 11:58:14,542 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:58:14,542 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:14,542 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:14,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:14,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:14,542 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:58:14,543 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:58:14,543 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,543 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:58:14,543 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:58:14,543 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,543 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:58:14,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:14,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:14,543 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:14,543 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:58:14,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:14,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:14,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:14,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:14,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:14,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:58:14,545 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:58:14,545 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:58:14,545 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:58:14,545 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:58:14,545 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:58:14,545 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:58:14,546 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:14,546 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:14,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:14,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:14,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:14,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:14,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:14,546 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:14,546 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:14,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:14,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:14,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:14,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:14,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:14,548 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:14,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:14,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:14,549 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,549 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:14,549 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:14,549 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:14,549 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:14,549 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:58:14,549 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:58:14,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:14,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:14,549 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:14,550 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,550 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:14,550 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:14,550 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:14,550 DEBUG [Thread-3389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:58:14,550 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:58:14,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:14,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:14,550 DEBUG [Thread-3389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:58:14,576 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=99 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/B/ec0c96c278464a37b27af8afea396d1e 2023-03-21 11:58:14,650 DEBUG [Thread-3389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:58:14,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:58:14,652 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:58:14,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:58:14,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:58:14,653 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:58:14,653 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:14,653 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:58:14,653 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:58:14,653 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:14,653 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:14,654 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:14,654 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:14,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:14,654 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,654 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:58:14,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:14,654 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:14,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:58:14,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:14,655 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:58:14,655 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:58:14,655 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:58:14,655 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:58:14,655 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:14,655 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:58:14,655 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:58:14,655 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:58:14,655 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:14,655 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. started... 2023-03-21 11:58:14,655 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. as already flushing 2023-03-21 11:58:14,655 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:14,655 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:58:14,655 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:58:14,655 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,655 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:58:14,655 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:58:14,656 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,656 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:14,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:14,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:14,656 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:14,656 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:58:14,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:14,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:14,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:14,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:14,657 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:58:14,657 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3c534adc[Count = 0] remaining members to acquire global barrier 2023-03-21 11:58:14,657 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:58:14,657 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:14,658 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:14,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:14,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:14,658 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:58:14,658 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:58:14,658 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:58:14,658 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,658 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:58:14,673 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,673 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:58:14,674 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:14,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:14,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:14,674 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:58:14,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:14,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:14,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:14,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:14,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:14,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:58:14,675 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:58:14,675 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:58:14,675 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:58:14,675 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:58:14,675 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:58:14,676 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:58:14,677 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:14,677 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:14,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:14,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:14,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:14,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:14,677 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:14,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:14,678 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:14,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:14,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:14,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:14,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:14,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:14,679 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:14,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:14,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:14,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,680 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,680 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:14,680 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:14,680 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:58:14,680 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:58:14,680 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:14,680 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:14,680 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:14,680 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,680 DEBUG [Thread-3389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:58:14,680 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:58:14,681 DEBUG [Thread-3389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:58:14,680 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:14,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:14,681 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:14,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:14,681 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:14,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:14,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:14,781 DEBUG [Thread-3389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:58:14,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:58:14,782 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:58:14,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:58:14,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:58:14,783 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:58:14,783 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:14,783 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:58:14,783 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:58:14,784 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:14,784 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:14,784 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:14,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:14,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:14,785 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:14,785 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,785 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:58:14,785 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:14,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:58:14,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:14,785 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:58:14,785 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:58:14,785 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:58:14,785 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:58:14,785 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:14,786 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:58:14,786 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:58:14,786 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:58:14,786 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:14,786 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. started... 2023-03-21 11:58:14,786 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. as already flushing 2023-03-21 11:58:14,786 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:14,786 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:58:14,786 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:58:14,786 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,786 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:58:14,786 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:58:14,787 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:14,787 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:14,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:14,787 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:14,787 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:58:14,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:14,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:14,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:14,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:14,788 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:58:14,789 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@840f337[Count = 0] remaining members to acquire global barrier 2023-03-21 11:58:14,789 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:58:14,789 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:14,789 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:14,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:14,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:14,789 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:58:14,789 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,789 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:58:14,789 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:58:14,789 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:58:14,790 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:58:14,790 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:14,790 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,790 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:58:14,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:14,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:14,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:14,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:14,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:14,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:14,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:14,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:58:14,792 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:58:14,792 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:58:14,792 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:58:14,792 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:58:14,792 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:58:14,792 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:58:14,793 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:14,793 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:14,793 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:14,793 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:14,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:14,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:14,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:14,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:14,793 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,793 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:14,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:14,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:14,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:14,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:14,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:14,794 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:14,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:14,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:14,795 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,795 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:14,795 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:14,795 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:58:14,796 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:58:14,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:14,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:14,795 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:14,795 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:14,796 DEBUG [Thread-3389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:58:14,796 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:14,796 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,796 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:14,796 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:14,796 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:14,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:14,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:14,796 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:58:14,796 DEBUG [Thread-3389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:58:14,896 DEBUG [Thread-3389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:58:14,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:58:14,898 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:58:14,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:58:14,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:58:14,899 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:58:14,899 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:14,899 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:58:14,899 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:58:14,900 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:14,900 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:14,900 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:14,900 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:14,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:14,901 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:14,901 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:58:14,901 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:14,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:58:14,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:14,901 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:58:14,901 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:58:14,901 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:58:14,901 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:58:14,901 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:14,901 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:58:14,902 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:58:14,902 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:58:14,902 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:14,902 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. started... 2023-03-21 11:58:14,902 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. as already flushing 2023-03-21 11:58:14,902 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:14,902 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:58:14,902 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:58:14,902 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,902 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:58:14,902 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:58:14,903 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,903 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:14,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:14,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:14,903 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:14,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:14,903 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:58:14,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:14,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:14,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:14,904 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:58:14,904 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7705215[Count = 0] remaining members to acquire global barrier 2023-03-21 11:58:14,904 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:58:14,904 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:14,904 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:14,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:14,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:14,904 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,905 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:58:14,905 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:58:14,905 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:58:14,905 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:58:14,906 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:58:14,906 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:14,906 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,906 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:58:14,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:14,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:14,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:14,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:14,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:14,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:14,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:14,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:58:14,908 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:58:14,908 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:58:14,908 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:58:14,908 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:58:14,908 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:58:14,908 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:58:14,909 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:14,909 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:14,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:14,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:14,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:14,909 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:14,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:14,909 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:14,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:14,909 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:14,909 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:14,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:14,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:14,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:14,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:14,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,910 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:14,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:14,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,912 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:14,912 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:14,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:14,912 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:58:14,912 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:58:14,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:14,912 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,912 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:14,912 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:14,912 DEBUG [Thread-3389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:58:14,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:14,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:14,912 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:14,912 DEBUG [Thread-3389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:58:14,912 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:14,912 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:14,912 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:14,912 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:14,983 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=99 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/C/6823865344744aed969482e603e43fa9 2023-03-21 11:58:14,987 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/A/41db545ed1c544c88c15c6ce47280df2 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/41db545ed1c544c88c15c6ce47280df2 2023-03-21 11:58:14,989 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/41db545ed1c544c88c15c6ce47280df2, entries=300, sequenceid=99, filesize=45.5 K 2023-03-21 11:58:14,990 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/B/ec0c96c278464a37b27af8afea396d1e as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/ec0c96c278464a37b27af8afea396d1e 2023-03-21 11:58:14,992 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/ec0c96c278464a37b27af8afea396d1e, entries=150, sequenceid=99, filesize=11.6 K 2023-03-21 11:58:14,993 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/C/6823865344744aed969482e603e43fa9 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/6823865344744aed969482e603e43fa9 2023-03-21 11:58:14,995 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/6823865344744aed969482e603e43fa9, entries=150, sequenceid=99, filesize=11.6 K 2023-03-21 11:58:14,995 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for cc711a8c4e0d80e60aabbf61e79ab961 in 1641ms, sequenceid=99, compaction requested=true 2023-03-21 11:58:14,995 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for cc711a8c4e0d80e60aabbf61e79ab961: 2023-03-21 11:58:14,996 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-21 11:58:14,996 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-21 11:58:14,996 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-21 11:58:14,996 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-21 11:58:14,996 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-21 11:58:14,997 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 123911 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-21 11:58:14,997 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1912): cc711a8c4e0d80e60aabbf61e79ab961/A is initiating minor compaction (all files) 2023-03-21 11:58:14,997 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of cc711a8c4e0d80e60aabbf61e79ab961/A in TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:14,997 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47647 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-21 11:58:14,997 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/92d9338ebb4740c481ddd1dcf985d71d, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/fe27dff1472d4ce59d37bb5d641342f8, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/c486aa6e3be947f9aa1c212c84045f2a, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/41db545ed1c544c88c15c6ce47280df2] into tmpdir=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp, totalSize=121.0 K 2023-03-21 11:58:14,997 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1912): cc711a8c4e0d80e60aabbf61e79ab961/B is initiating minor compaction (all files) 2023-03-21 11:58:14,997 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegion(2259): Starting compaction of cc711a8c4e0d80e60aabbf61e79ab961/B in TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:14,997 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/62536a8df9444844907c95116eb74784, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/62399949fee641a395f900f5c357b5d9, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/f586393140554ad19d819ac9da0c00fe, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/ec0c96c278464a37b27af8afea396d1e] into tmpdir=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp, totalSize=46.5 K 2023-03-21 11:58:14,997 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 92d9338ebb4740c481ddd1dcf985d71d, keycount=150, bloomtype=ROW, size=25.2 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1679399890473 2023-03-21 11:58:14,997 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 62536a8df9444844907c95116eb74784, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1679399890473 2023-03-21 11:58:14,998 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting fe27dff1472d4ce59d37bb5d641342f8, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=72, earliestPutTs=1679399890593 2023-03-21 11:58:14,998 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 62399949fee641a395f900f5c357b5d9, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=72, earliestPutTs=1679399890593 2023-03-21 11:58:14,998 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting c486aa6e3be947f9aa1c212c84045f2a, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=86, earliestPutTs=1679399891209 2023-03-21 11:58:14,998 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting f586393140554ad19d819ac9da0c00fe, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=86, earliestPutTs=1679399891209 2023-03-21 11:58:14,998 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 41db545ed1c544c88c15c6ce47280df2, keycount=300, bloomtype=ROW, size=45.5 K, encoding=NONE, compression=NONE, seqNum=99, earliestPutTs=1679399893339 2023-03-21 11:58:14,998 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting ec0c96c278464a37b27af8afea396d1e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=99, earliestPutTs=1679399893341 2023-03-21 11:58:15,004 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] throttle.PressureAwareThroughputController(145): cc711a8c4e0d80e60aabbf61e79ab961#B#compaction#454 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 57.69 MB/second 2023-03-21 11:58:15,005 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] throttle.PressureAwareThroughputController(145): cc711a8c4e0d80e60aabbf61e79ab961#A#compaction#453 average throughput is 9.86 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 57.69 MB/second 2023-03-21 11:58:15,012 DEBUG [Thread-3389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:58:15,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:58:15,014 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:58:15,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:58:15,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:58:15,015 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:58:15,015 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:15,015 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:58:15,015 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:58:15,015 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:15,015 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:15,016 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:15,016 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:15,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:15,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:15,016 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:15,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:15,016 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:58:15,016 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:15,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:58:15,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:15,017 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:58:15,017 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:58:15,017 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:58:15,017 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:58:15,017 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:15,017 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:58:15,022 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:58:15,022 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:58:15,022 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:15,022 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. started... 2023-03-21 11:58:15,022 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing cc711a8c4e0d80e60aabbf61e79ab961 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-03-21 11:58:15,022 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cc711a8c4e0d80e60aabbf61e79ab961, store=A 2023-03-21 11:58:15,023 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-21 11:58:15,023 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cc711a8c4e0d80e60aabbf61e79ab961, store=B 2023-03-21 11:58:15,023 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-21 11:58:15,023 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cc711a8c4e0d80e60aabbf61e79ab961, store=C 2023-03-21 11:58:15,023 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-21 11:58:15,422 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/B/c7cb4612afbc4523b757d4c69a823c1b as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/c7cb4612afbc4523b757d4c69a823c1b 2023-03-21 11:58:15,425 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in cc711a8c4e0d80e60aabbf61e79ab961/B of cc711a8c4e0d80e60aabbf61e79ab961 into c7cb4612afbc4523b757d4c69a823c1b(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-21 11:58:15,425 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for cc711a8c4e0d80e60aabbf61e79ab961: 2023-03-21 11:58:15,425 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961., storeName=cc711a8c4e0d80e60aabbf61e79ab961/B, priority=12, startTime=1679399894996; duration=0sec 2023-03-21 11:58:15,425 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-21 11:58:15,425 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-21 11:58:15,426 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47647 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-21 11:58:15,426 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1912): cc711a8c4e0d80e60aabbf61e79ab961/C is initiating minor compaction (all files) 2023-03-21 11:58:15,426 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegion(2259): Starting compaction of cc711a8c4e0d80e60aabbf61e79ab961/C in TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:15,426 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/2491db6eee204fdd84376f13fb75c149, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/4bcb4e67df8c4d97b62f9c129502c526, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/6045696af9744b4f91b718d1213352fc, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/6823865344744aed969482e603e43fa9] into tmpdir=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp, totalSize=46.5 K 2023-03-21 11:58:15,427 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 2491db6eee204fdd84376f13fb75c149, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1679399890473 2023-03-21 11:58:15,427 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 4bcb4e67df8c4d97b62f9c129502c526, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=72, earliestPutTs=1679399890593 2023-03-21 11:58:15,427 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 6045696af9744b4f91b718d1213352fc, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=86, earliestPutTs=1679399891209 2023-03-21 11:58:15,427 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 6823865344744aed969482e603e43fa9, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=99, earliestPutTs=1679399893341 2023-03-21 11:58:15,432 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] throttle.PressureAwareThroughputController(145): cc711a8c4e0d80e60aabbf61e79ab961#C#compaction#456 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 57.69 MB/second 2023-03-21 11:58:15,439 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/A/f2c16c492d9046a396017eae634bbec2 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/f2c16c492d9046a396017eae634bbec2 2023-03-21 11:58:15,444 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in cc711a8c4e0d80e60aabbf61e79ab961/A of cc711a8c4e0d80e60aabbf61e79ab961 into f2c16c492d9046a396017eae634bbec2(size=25.4 K), total size for store is 25.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-21 11:58:15,444 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for cc711a8c4e0d80e60aabbf61e79ab961: 2023-03-21 11:58:15,444 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961., storeName=cc711a8c4e0d80e60aabbf61e79ab961/A, priority=12, startTime=1679399894996; duration=0sec 2023-03-21 11:58:15,444 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-21 11:58:15,445 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230321b6fbf3a7aaf645f9826c71a676328540 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230321b6fbf3a7aaf645f9826c71a676328540 2023-03-21 11:58:15,448 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=134, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/A/a98a48d8082a4ca1b951afa24ad40869 2023-03-21 11:58:15,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(9158): Flush requested on cc711a8c4e0d80e60aabbf61e79ab961 2023-03-21 11:58:15,497 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. as already flushing 2023-03-21 11:58:15,511 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:15,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55146 deadline: 1679399955508, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:15,511 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:15,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55112 deadline: 1679399955509, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:15,512 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:15,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55178 deadline: 1679399955509, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:15,515 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:15,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55130 deadline: 1679399955511, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:15,515 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:15,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55174 deadline: 1679399955512, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:15,612 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:15,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55146 deadline: 1679399955612, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:15,616 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:15,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55130 deadline: 1679399955616, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:15,815 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:15,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55146 deadline: 1679399955813, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:15,819 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:15,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55130 deadline: 1679399955818, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:15,840 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/C/9e5ec838908846808d55649fb1447353 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/9e5ec838908846808d55649fb1447353 2023-03-21 11:58:15,843 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in cc711a8c4e0d80e60aabbf61e79ab961/C of cc711a8c4e0d80e60aabbf61e79ab961 into 9e5ec838908846808d55649fb1447353(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-21 11:58:15,843 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for cc711a8c4e0d80e60aabbf61e79ab961: 2023-03-21 11:58:15,843 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961., storeName=cc711a8c4e0d80e60aabbf61e79ab961/C, priority=12, startTime=1679399894996; duration=0sec 2023-03-21 11:58:15,843 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-21 11:58:15,855 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=134 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/B/3e379f97af55485191c99a71435d9a11 2023-03-21 11:58:15,862 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=134 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/C/f9fd1189769a4e55a7b416911e3bb8a3 2023-03-21 11:58:15,865 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/A/a98a48d8082a4ca1b951afa24ad40869 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/a98a48d8082a4ca1b951afa24ad40869 2023-03-21 11:58:15,867 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/a98a48d8082a4ca1b951afa24ad40869, entries=150, sequenceid=134, filesize=25.3 K 2023-03-21 11:58:15,868 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/B/3e379f97af55485191c99a71435d9a11 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/3e379f97af55485191c99a71435d9a11 2023-03-21 11:58:15,871 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/3e379f97af55485191c99a71435d9a11, entries=150, sequenceid=134, filesize=11.8 K 2023-03-21 11:58:15,871 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/C/f9fd1189769a4e55a7b416911e3bb8a3 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/f9fd1189769a4e55a7b416911e3bb8a3 2023-03-21 11:58:15,874 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/f9fd1189769a4e55a7b416911e3bb8a3, entries=150, sequenceid=134, filesize=11.8 K 2023-03-21 11:58:15,874 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=53.67 KB/54960 for cc711a8c4e0d80e60aabbf61e79ab961 in 852ms, sequenceid=134, compaction requested=false 2023-03-21 11:58:15,875 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for cc711a8c4e0d80e60aabbf61e79ab961: 2023-03-21 11:58:15,875 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:15,875 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:58:15,875 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:58:15,875 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:15,875 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:58:15,875 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:58:15,876 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:15,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:15,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:15,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:15,876 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:15,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:15,876 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:15,876 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:58:15,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:15,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:15,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:15,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:15,877 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:58:15,877 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c864129[Count = 0] remaining members to acquire global barrier 2023-03-21 11:58:15,877 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:58:15,877 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:15,878 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:15,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:15,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:15,878 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:58:15,878 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:58:15,878 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:58:15,878 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:15,878 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:58:15,878 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:15,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:15,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:15,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:15,879 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:58:15,879 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:15,879 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:58:15,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:15,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:15,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:15,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:15,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:15,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:15,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:15,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:58:15,881 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:58:15,881 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:58:15,881 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:58:15,881 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:58:15,881 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:58:15,881 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:58:15,882 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:15,882 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:15,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:15,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:15,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:15,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:15,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:15,882 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:15,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:15,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:15,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:15,883 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:15,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:15,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:15,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:15,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:15,884 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:15,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:15,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:15,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:15,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:15,885 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:15,885 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:58:15,885 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:58:15,885 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:15,885 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:58:15,885 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:15,885 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:15,885 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:15,885 DEBUG [Thread-3389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:58:15,885 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:15,886 DEBUG [Thread-3389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:58:15,886 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:15,886 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:15,886 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:15,886 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:15,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:15,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:15,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:15,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:15,986 DEBUG [Thread-3389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:58:15,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:58:15,987 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:58:15,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:58:15,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:58:15,988 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:58:15,988 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:15,988 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:58:15,988 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:58:15,989 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:15,989 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:15,989 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:15,989 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:15,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:15,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:15,989 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:15,989 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:58:15,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:15,990 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:15,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:58:15,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:15,990 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:58:15,990 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:58:15,990 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:58:15,990 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:58:15,990 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:15,990 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:58:15,990 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:58:15,990 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:58:15,990 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:15,990 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. started... 2023-03-21 11:58:15,990 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing cc711a8c4e0d80e60aabbf61e79ab961 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-21 11:58:15,991 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cc711a8c4e0d80e60aabbf61e79ab961, store=A 2023-03-21 11:58:15,991 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:58:15,991 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cc711a8c4e0d80e60aabbf61e79ab961, store=B 2023-03-21 11:58:15,991 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:58:15,991 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cc711a8c4e0d80e60aabbf61e79ab961, store=C 2023-03-21 11:58:15,991 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:58:16,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(9158): Flush requested on cc711a8c4e0d80e60aabbf61e79ab961 2023-03-21 11:58:16,118 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. as already flushing 2023-03-21 11:58:16,140 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:16,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55146 deadline: 1679399956138, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:16,141 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679399896132, max timestamp=1679399896140Num uniques 859; 2023-03-21 11:58:16,141 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-21 11:58:16,142 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679399896132, max timestamp=1679399896140Num uniques 859; 2023-03-21 11:58:16,142 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-21 11:58:16,142 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679399896132, max timestamp=1679399896140Num uniques 859; 2023-03-21 11:58:16,142 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-21 11:58:16,197 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:16,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55130 deadline: 1679399956195, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:16,244 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:16,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55146 deadline: 1679399956242, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:16,298 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:16,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55130 deadline: 1679399956297, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:16,402 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303214f25b8a27ef04386bbdab5edb13a9e2a to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303214f25b8a27ef04386bbdab5edb13a9e2a 2023-03-21 11:58:16,407 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=148, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/A/b3bcf43bd78440f882303558ec4be526 2023-03-21 11:58:16,447 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:16,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55146 deadline: 1679399956446, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:16,501 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:16,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55130 deadline: 1679399956499, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:16,751 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:16,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55146 deadline: 1679399956749, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:16,805 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:16,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55130 deadline: 1679399956804, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:16,813 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=148 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/B/e8b79d7e121f4c82b00768a168e3334f 2023-03-21 11:58:16,820 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=148 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/C/260e444f57bc496eaa872ac7cfaaf58d 2023-03-21 11:58:16,822 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/A/b3bcf43bd78440f882303558ec4be526 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/b3bcf43bd78440f882303558ec4be526 2023-03-21 11:58:16,825 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/b3bcf43bd78440f882303558ec4be526, entries=150, sequenceid=148, filesize=25.3 K 2023-03-21 11:58:16,826 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/B/e8b79d7e121f4c82b00768a168e3334f as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/e8b79d7e121f4c82b00768a168e3334f 2023-03-21 11:58:16,828 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/e8b79d7e121f4c82b00768a168e3334f, entries=150, sequenceid=148, filesize=11.8 K 2023-03-21 11:58:16,828 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/C/260e444f57bc496eaa872ac7cfaaf58d as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/260e444f57bc496eaa872ac7cfaaf58d 2023-03-21 11:58:16,831 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/260e444f57bc496eaa872ac7cfaaf58d, entries=150, sequenceid=148, filesize=11.8 K 2023-03-21 11:58:16,831 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for cc711a8c4e0d80e60aabbf61e79ab961 in 841ms, sequenceid=148, compaction requested=true 2023-03-21 11:58:16,831 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for cc711a8c4e0d80e60aabbf61e79ab961: 2023-03-21 11:58:16,831 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:16,831 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:58:16,831 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:58:16,831 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:16,831 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:58:16,831 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:58:16,832 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:16,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:16,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:16,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:16,832 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:16,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:16,833 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:16,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:16,833 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:58:16,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:16,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:16,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:16,834 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:58:16,834 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a25f1cf[Count = 0] remaining members to acquire global barrier 2023-03-21 11:58:16,834 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:58:16,834 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:16,834 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:16,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:16,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:16,835 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:58:16,835 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:58:16,835 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:16,835 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:58:16,835 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:58:16,835 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:58:16,835 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:16,835 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:16,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:16,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:16,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:16,835 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:58:16,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:16,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:16,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:16,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:16,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:16,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:16,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:16,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:58:16,837 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:58:16,837 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:58:16,837 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:58:16,838 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:58:16,838 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:58:16,838 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:58:16,839 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:16,839 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:16,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:16,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:16,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:16,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:16,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:16,839 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:16,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:16,839 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:16,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:16,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:16,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:16,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:16,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:16,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:16,840 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:16,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:16,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:16,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:16,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:16,842 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:16,842 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:16,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:16,842 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:16,842 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:16,842 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:58:16,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:16,842 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:58:16,842 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:58:16,842 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:16,842 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:16,842 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:16,842 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:16,842 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:16,842 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:16,842 DEBUG [Thread-3389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:58:16,842 DEBUG [Thread-3389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:58:16,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:16,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:16,942 DEBUG [Thread-3389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:58:16,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:58:16,944 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:58:16,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:58:16,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:58:16,945 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:58:16,945 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:16,945 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:58:16,945 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:58:16,946 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:16,946 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:16,946 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:16,946 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:16,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:16,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:16,947 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:16,947 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:58:16,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:16,947 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:16,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:58:16,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:16,947 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:58:16,947 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:58:16,947 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:58:16,948 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:58:16,948 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:16,948 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:58:16,948 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:58:16,948 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:58:16,948 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:16,948 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. started... 2023-03-21 11:58:16,948 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing cc711a8c4e0d80e60aabbf61e79ab961 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-03-21 11:58:16,948 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cc711a8c4e0d80e60aabbf61e79ab961, store=A 2023-03-21 11:58:16,948 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-21 11:58:16,948 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cc711a8c4e0d80e60aabbf61e79ab961, store=B 2023-03-21 11:58:16,948 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-21 11:58:16,949 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cc711a8c4e0d80e60aabbf61e79ab961, store=C 2023-03-21 11:58:16,949 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-21 11:58:17,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(9158): Flush requested on cc711a8c4e0d80e60aabbf61e79ab961 2023-03-21 11:58:17,254 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. as already flushing 2023-03-21 11:58:17,277 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:17,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55146 deadline: 1679399957273, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:17,308 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:17,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55130 deadline: 1679399957307, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:17,358 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303219561cf4b1cc24e8493c2c1ddebe6ef34 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303219561cf4b1cc24e8493c2c1ddebe6ef34 2023-03-21 11:58:17,379 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:17,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55146 deadline: 1679399957378, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:17,527 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:17,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55112 deadline: 1679399957525, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:17,529 DEBUG [Thread-3387] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4137 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961., hostname=jenkins-hbase20.apache.org,34369,1679399732030, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-21 11:58:17,531 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:17,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55178 deadline: 1679399957529, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:17,532 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:17,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55174 deadline: 1679399957529, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:17,533 DEBUG [Thread-3383] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4139 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961., hostname=jenkins-hbase20.apache.org,34369,1679399732030, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-21 11:58:17,534 DEBUG [Thread-3381] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4142 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961., hostname=jenkins-hbase20.apache.org,34369,1679399732030, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-21 11:58:17,583 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:17,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55146 deadline: 1679399957580, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:17,762 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=184, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/A/defb923ec42b470cb29d18cd54b28bd4 2023-03-21 11:58:17,885 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:17,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55146 deadline: 1679399957883, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:18,170 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=184 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/B/8918745ddf7e412d986b8d3b68703819 2023-03-21 11:58:18,178 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=184 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/C/57d09b7c6c164e71b25efd5aa897f8f1 2023-03-21 11:58:18,181 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/A/defb923ec42b470cb29d18cd54b28bd4 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/defb923ec42b470cb29d18cd54b28bd4 2023-03-21 11:58:18,183 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/defb923ec42b470cb29d18cd54b28bd4, entries=150, sequenceid=184, filesize=25.3 K 2023-03-21 11:58:18,184 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/B/8918745ddf7e412d986b8d3b68703819 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/8918745ddf7e412d986b8d3b68703819 2023-03-21 11:58:18,186 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/8918745ddf7e412d986b8d3b68703819, entries=150, sequenceid=184, filesize=11.8 K 2023-03-21 11:58:18,186 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/C/57d09b7c6c164e71b25efd5aa897f8f1 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/57d09b7c6c164e71b25efd5aa897f8f1 2023-03-21 11:58:18,189 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/57d09b7c6c164e71b25efd5aa897f8f1, entries=150, sequenceid=184, filesize=11.8 K 2023-03-21 11:58:18,189 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=53.67 KB/54960 for cc711a8c4e0d80e60aabbf61e79ab961 in 1241ms, sequenceid=184, compaction requested=true 2023-03-21 11:58:18,189 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for cc711a8c4e0d80e60aabbf61e79ab961: 2023-03-21 11:58:18,189 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:18,190 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:58:18,190 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:58:18,190 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:18,190 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:58:18,190 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:58:18,191 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:18,191 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:18,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:18,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:18,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:18,191 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:18,191 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:58:18,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:18,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:18,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:18,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:18,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:18,192 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:58:18,192 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@557368f7[Count = 0] remaining members to acquire global barrier 2023-03-21 11:58:18,192 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:58:18,192 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:18,195 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:18,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:18,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:18,196 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:58:18,196 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:58:18,196 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:18,196 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:58:18,196 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:58:18,196 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:18,197 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:58:18,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:18,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:18,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:18,197 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:18,197 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:58:18,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:18,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:18,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:18,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:18,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:18,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:18,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:18,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:58:18,200 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:58:18,200 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:58:18,200 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:58:18,200 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:58:18,200 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:58:18,200 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:58:18,201 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:18,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:18,201 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:18,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:18,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:18,201 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:18,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:18,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:18,201 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:18,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:18,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:18,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:18,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:18,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:18,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:18,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:18,202 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:18,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:18,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:18,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:18,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:18,204 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:18,204 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:18,204 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:18,204 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:58:18,204 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:18,204 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:58:18,204 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:58:18,204 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:18,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:18,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:18,204 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:18,204 DEBUG [Thread-3389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:58:18,204 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:18,204 DEBUG [Thread-3389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:58:18,205 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:18,205 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:18,205 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:18,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:18,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:18,305 DEBUG [Thread-3389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:58:18,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:58:18,307 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:58:18,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:58:18,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:58:18,308 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:58:18,308 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:18,308 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:58:18,308 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:58:18,308 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:18,308 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:18,308 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:18,309 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:18,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:18,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:18,309 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:18,309 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:58:18,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:18,309 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:18,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:58:18,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:18,309 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:58:18,310 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:58:18,310 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:58:18,310 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:58:18,310 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:18,310 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:58:18,310 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:58:18,310 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:58:18,310 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:18,310 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. started... 2023-03-21 11:58:18,310 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing cc711a8c4e0d80e60aabbf61e79ab961 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-21 11:58:18,310 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cc711a8c4e0d80e60aabbf61e79ab961, store=A 2023-03-21 11:58:18,310 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:58:18,310 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cc711a8c4e0d80e60aabbf61e79ab961, store=B 2023-03-21 11:58:18,310 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:58:18,311 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cc711a8c4e0d80e60aabbf61e79ab961, store=C 2023-03-21 11:58:18,311 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:58:18,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(9158): Flush requested on cc711a8c4e0d80e60aabbf61e79ab961 2023-03-21 11:58:18,314 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. as already flushing 2023-03-21 11:58:18,323 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230321cd9d22d502084078ab05a7a1a761fe80 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230321cd9d22d502084078ab05a7a1a761fe80 2023-03-21 11:58:18,373 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679399898342, max timestamp=1679399898372Num uniques 859; 2023-03-21 11:58:18,373 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-21 11:58:18,373 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679399898342, max timestamp=1679399898372Num uniques 859; 2023-03-21 11:58:18,373 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-21 11:58:18,374 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679399898342, max timestamp=1679399898372Num uniques 859; 2023-03-21 11:58:18,374 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-21 11:58:18,404 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:18,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55130 deadline: 1679399958402, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:18,407 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:18,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55146 deadline: 1679399958405, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:18,507 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:18,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55130 deadline: 1679399958505, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:18,510 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:18,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55146 deadline: 1679399958508, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:18,711 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:18,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55130 deadline: 1679399958709, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:18,713 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:18,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55146 deadline: 1679399958711, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:18,728 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=195, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/A/292986e6656c4c3693e3a2f2f600a7d2 2023-03-21 11:58:19,013 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:19,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55130 deadline: 1679399959012, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:19,018 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:19,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55146 deadline: 1679399959014, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:19,135 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=195 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/B/c6c884a4724648c88009f1b520daa5a1 2023-03-21 11:58:19,143 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=195 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/C/e46a05e063434332b9847c59242eba44 2023-03-21 11:58:19,146 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/A/292986e6656c4c3693e3a2f2f600a7d2 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/292986e6656c4c3693e3a2f2f600a7d2 2023-03-21 11:58:19,149 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/292986e6656c4c3693e3a2f2f600a7d2, entries=150, sequenceid=195, filesize=25.3 K 2023-03-21 11:58:19,149 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/B/c6c884a4724648c88009f1b520daa5a1 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/c6c884a4724648c88009f1b520daa5a1 2023-03-21 11:58:19,152 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/c6c884a4724648c88009f1b520daa5a1, entries=150, sequenceid=195, filesize=11.8 K 2023-03-21 11:58:19,153 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/C/e46a05e063434332b9847c59242eba44 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/e46a05e063434332b9847c59242eba44 2023-03-21 11:58:19,156 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/e46a05e063434332b9847c59242eba44, entries=150, sequenceid=195, filesize=11.8 K 2023-03-21 11:58:19,156 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for cc711a8c4e0d80e60aabbf61e79ab961 in 846ms, sequenceid=195, compaction requested=true 2023-03-21 11:58:19,156 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for cc711a8c4e0d80e60aabbf61e79ab961: 2023-03-21 11:58:19,156 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:19,156 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:58:19,156 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:58:19,156 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:19,156 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:58:19,156 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:58:19,157 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:19,157 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:19,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:19,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:19,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:19,157 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:19,158 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:58:19,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:19,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:19,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:19,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:19,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:19,158 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:58:19,159 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@501668b6[Count = 0] remaining members to acquire global barrier 2023-03-21 11:58:19,159 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:58:19,159 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:19,159 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:19,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:19,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:19,159 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:58:19,159 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:58:19,159 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:58:19,159 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:19,159 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:58:19,160 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:58:19,160 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:19,160 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:19,160 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:58:19,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:19,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:19,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:19,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:19,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:19,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:19,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:19,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:19,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:19,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:19,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:58:19,162 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:58:19,162 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:58:19,162 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:58:19,162 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:58:19,162 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:58:19,162 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:58:19,163 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:19,163 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:19,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:19,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:19,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:19,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:19,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:19,163 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:19,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:19,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:19,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:19,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:19,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:19,164 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:19,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:19,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:19,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:19,165 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:19,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:19,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:19,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:19,166 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:19,166 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:19,166 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:19,166 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:19,166 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:19,166 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:19,166 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:19,166 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:58:19,166 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:58:19,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:19,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:19,166 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:19,166 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:19,166 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:19,166 DEBUG [Thread-3389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:58:19,166 DEBUG [Thread-3389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:58:19,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:19,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:19,266 DEBUG [Thread-3389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:58:19,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:58:19,268 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:58:19,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:58:19,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:58:19,269 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:58:19,269 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:19,269 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:58:19,269 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:58:19,270 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:19,270 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:19,270 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:19,270 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:19,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:19,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:19,271 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:19,271 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:58:19,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:19,271 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:19,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:58:19,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:19,271 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:58:19,271 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:58:19,271 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:58:19,272 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:58:19,272 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:19,272 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:58:19,272 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:58:19,272 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:58:19,272 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:19,272 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. started... 2023-03-21 11:58:19,272 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing cc711a8c4e0d80e60aabbf61e79ab961 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-03-21 11:58:19,272 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cc711a8c4e0d80e60aabbf61e79ab961, store=A 2023-03-21 11:58:19,272 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-21 11:58:19,272 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cc711a8c4e0d80e60aabbf61e79ab961, store=B 2023-03-21 11:58:19,272 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-21 11:58:19,272 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cc711a8c4e0d80e60aabbf61e79ab961, store=C 2023-03-21 11:58:19,272 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-21 11:58:19,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(9158): Flush requested on cc711a8c4e0d80e60aabbf61e79ab961 2023-03-21 11:58:19,519 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. as already flushing 2023-03-21 11:58:19,534 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:19,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55130 deadline: 1679399959530, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:19,538 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:19,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55146 deadline: 1679399959533, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:19,636 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:19,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55130 deadline: 1679399959635, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:19,641 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:19,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55146 deadline: 1679399959639, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:19,686 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230321bb5cf131faac429e91293dff402d4147 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230321bb5cf131faac429e91293dff402d4147 2023-03-21 11:58:19,689 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=231, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/A/5c23338f1a8240e08d210963d0f06716 2023-03-21 11:58:19,840 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:19,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55130 deadline: 1679399959838, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:19,844 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:19,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55146 deadline: 1679399959842, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,098 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=231 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/B/af95ed8c42ca478cb3e7dce00bc46237 2023-03-21 11:58:20,105 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=231 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/C/47ffd81cd1cf481d865ab08a2d1330b9 2023-03-21 11:58:20,109 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/A/5c23338f1a8240e08d210963d0f06716 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/5c23338f1a8240e08d210963d0f06716 2023-03-21 11:58:20,111 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/5c23338f1a8240e08d210963d0f06716, entries=150, sequenceid=231, filesize=25.3 K 2023-03-21 11:58:20,112 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/B/af95ed8c42ca478cb3e7dce00bc46237 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/af95ed8c42ca478cb3e7dce00bc46237 2023-03-21 11:58:20,114 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/af95ed8c42ca478cb3e7dce00bc46237, entries=150, sequenceid=231, filesize=11.8 K 2023-03-21 11:58:20,115 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/C/47ffd81cd1cf481d865ab08a2d1330b9 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/47ffd81cd1cf481d865ab08a2d1330b9 2023-03-21 11:58:20,117 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/47ffd81cd1cf481d865ab08a2d1330b9, entries=150, sequenceid=231, filesize=11.8 K 2023-03-21 11:58:20,117 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=53.67 KB/54960 for cc711a8c4e0d80e60aabbf61e79ab961 in 845ms, sequenceid=231, compaction requested=true 2023-03-21 11:58:20,118 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for cc711a8c4e0d80e60aabbf61e79ab961: 2023-03-21 11:58:20,118 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:20,118 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:58:20,118 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:58:20,118 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,118 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:58:20,118 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:58:20,119 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:20,119 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:20,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:20,119 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:20,119 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:58:20,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:20,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:20,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:20,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:20,120 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:58:20,120 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e57e494[Count = 0] remaining members to acquire global barrier 2023-03-21 11:58:20,120 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:58:20,120 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:20,120 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:20,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:20,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:20,121 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:58:20,121 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:58:20,121 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:58:20,121 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,121 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:58:20,121 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:20,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:20,121 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:58:20,121 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:20,121 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:58:20,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:20,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:20,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:20,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:20,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:20,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:58:20,123 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:58:20,123 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:58:20,123 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:58:20,123 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:58:20,123 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:58:20,123 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:58:20,124 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:20,124 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:20,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:20,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:20,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:20,124 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:20,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:20,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:20,124 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:20,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:20,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:20,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:20,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:20,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:20,125 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:20,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:20,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:20,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,126 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,126 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:20,126 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:20,127 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:20,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:20,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:20,127 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:20,127 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:58:20,127 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:58:20,127 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:20,127 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:20,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:20,127 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:58:20,127 DEBUG [Thread-3389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:58:20,127 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:20,127 DEBUG [Thread-3389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:58:20,127 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:20,127 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:20,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(9158): Flush requested on cc711a8c4e0d80e60aabbf61e79ab961 2023-03-21 11:58:20,145 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing cc711a8c4e0d80e60aabbf61e79ab961 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-21 11:58:20,145 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cc711a8c4e0d80e60aabbf61e79ab961, store=A 2023-03-21 11:58:20,145 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:58:20,145 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cc711a8c4e0d80e60aabbf61e79ab961, store=B 2023-03-21 11:58:20,145 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:58:20,145 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cc711a8c4e0d80e60aabbf61e79ab961, store=C 2023-03-21 11:58:20,145 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:58:20,185 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:20,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55130 deadline: 1679399960181, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,187 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:20,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55146 deadline: 1679399960184, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,227 DEBUG [Thread-3389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:58:20,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:58:20,229 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:58:20,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:58:20,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:58:20,234 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:58:20,234 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:20,234 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:58:20,234 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:58:20,234 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:20,234 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:20,235 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:20,235 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:20,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:20,235 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,235 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:58:20,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:20,236 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:20,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:58:20,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:20,236 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:58:20,236 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:58:20,236 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:58:20,236 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:58:20,236 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:20,236 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:58:20,236 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:58:20,236 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:58:20,236 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:20,237 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. started... 2023-03-21 11:58:20,237 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. as already flushing 2023-03-21 11:58:20,237 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:20,237 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:58:20,237 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:58:20,237 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,237 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:58:20,237 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:58:20,238 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,238 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:20,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:20,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:20,238 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:20,238 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:58:20,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:20,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:20,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:20,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:20,239 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:58:20,239 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6b89d518[Count = 0] remaining members to acquire global barrier 2023-03-21 11:58:20,239 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:58:20,239 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:20,239 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:20,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:20,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:20,239 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:58:20,239 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:58:20,239 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:58:20,240 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,240 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:58:20,240 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,240 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:58:20,240 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:20,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:20,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:20,240 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:58:20,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:20,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:20,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:20,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:20,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:20,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:58:20,242 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:58:20,242 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:58:20,242 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:58:20,242 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:58:20,242 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:58:20,242 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:58:20,243 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:20,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:20,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:20,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:20,243 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:20,243 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:20,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:20,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:20,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:20,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:20,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:20,243 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:20,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:20,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:20,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:20,244 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:20,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:20,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,246 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,246 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:20,246 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:20,246 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:20,247 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:20,247 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,247 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:20,247 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:20,247 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:20,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:20,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:20,246 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:58:20,246 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:58:20,247 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:58:20,246 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:20,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:20,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:20,247 DEBUG [Thread-3389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:58:20,247 DEBUG [Thread-3389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:58:20,288 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:20,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55130 deadline: 1679399960286, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,290 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:20,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55146 deadline: 1679399960288, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,347 DEBUG [Thread-3389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:58:20,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:58:20,349 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:58:20,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:58:20,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:58:20,350 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:58:20,350 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:20,350 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:58:20,350 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:58:20,351 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:20,351 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:20,351 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:20,351 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:20,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:20,351 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,352 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:58:20,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:20,352 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:20,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:58:20,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:20,352 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:58:20,352 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:58:20,352 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:58:20,352 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:58:20,352 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:20,352 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:58:20,352 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:58:20,352 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:58:20,353 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:20,353 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. started... 2023-03-21 11:58:20,353 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. as already flushing 2023-03-21 11:58:20,353 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:20,353 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:58:20,353 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:58:20,353 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,353 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:58:20,353 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:58:20,354 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:20,354 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:20,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:20,354 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:20,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:20,354 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:58:20,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:20,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:20,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:20,355 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:58:20,355 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@25f32c4e[Count = 0] remaining members to acquire global barrier 2023-03-21 11:58:20,355 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:58:20,355 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:20,355 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:20,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:20,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:20,356 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:58:20,356 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,356 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:58:20,356 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:58:20,356 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:58:20,356 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,356 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:58:20,357 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:20,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:20,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:20,357 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:58:20,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:20,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:20,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:20,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:20,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:20,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:58:20,358 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:58:20,358 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:58:20,358 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:58:20,358 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:58:20,358 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:58:20,358 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:58:20,359 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:20,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:20,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:20,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:20,359 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:20,359 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:20,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:20,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:20,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:20,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:20,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:20,360 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:20,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:20,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:20,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:20,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,361 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:20,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:20,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,362 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,362 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:20,362 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:20,362 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,362 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:58:20,362 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:58:20,363 DEBUG [Thread-3389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:58:20,363 DEBUG [Thread-3389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:58:20,362 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:20,362 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:20,362 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:20,363 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:20,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:20,363 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:20,363 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:20,363 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:20,363 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:20,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:20,463 DEBUG [Thread-3389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:58:20,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:58:20,464 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:58:20,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:58:20,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:58:20,466 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:58:20,466 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:20,466 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:58:20,466 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:58:20,466 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:20,466 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:20,467 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:20,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:20,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:20,467 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:20,467 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,467 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:58:20,467 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:20,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:58:20,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:20,468 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:58:20,468 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:58:20,468 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:58:20,468 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:58:20,468 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:20,468 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:58:20,469 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:58:20,469 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:58:20,469 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:20,469 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. started... 2023-03-21 11:58:20,469 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. as already flushing 2023-03-21 11:58:20,469 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:20,469 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:58:20,469 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:58:20,469 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,469 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:58:20,469 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:58:20,470 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,470 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:20,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:20,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:20,470 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:20,470 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:58:20,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:20,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:20,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:20,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:20,471 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:58:20,471 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4643b39e[Count = 0] remaining members to acquire global barrier 2023-03-21 11:58:20,471 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:58:20,471 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:20,472 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:20,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:20,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:20,472 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:58:20,472 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,472 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:58:20,472 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:58:20,472 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:58:20,473 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:58:20,473 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,473 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:20,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:20,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:20,473 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:58:20,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:20,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:20,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:20,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:20,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:20,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:58:20,475 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:58:20,475 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:58:20,475 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:58:20,475 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:58:20,475 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:58:20,475 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:58:20,476 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:20,476 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:20,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:20,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:20,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:20,476 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:20,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:20,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:20,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:20,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:20,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:20,476 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:20,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:20,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:20,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:20,477 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:20,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:20,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,478 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,479 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:20,479 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:20,479 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:20,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:20,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:20,479 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:20,479 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:58:20,479 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:58:20,479 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:58:20,479 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:20,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:20,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:20,479 DEBUG [Thread-3389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:58:20,479 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,479 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:20,479 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:20,479 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:20,479 DEBUG [Thread-3389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:58:20,491 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:20,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55130 deadline: 1679399960489, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,492 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:20,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55146 deadline: 1679399960490, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,567 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230321c8fe639d648442afbca3bb0d312444dc to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230321c8fe639d648442afbca3bb0d312444dc 2023-03-21 11:58:20,579 DEBUG [Thread-3389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:58:20,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:58:20,581 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:58:20,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:58:20,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:58:20,582 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:58:20,582 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:20,582 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:58:20,582 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:58:20,583 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:20,583 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:20,583 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:20,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:20,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:20,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:20,584 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,584 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:20,584 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,584 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:58:20,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:58:20,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:20,584 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:58:20,584 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:58:20,584 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:58:20,584 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:58:20,584 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:20,584 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:58:20,585 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:58:20,585 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:58:20,585 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:20,585 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. started... 2023-03-21 11:58:20,585 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. as already flushing 2023-03-21 11:58:20,585 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:20,585 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:58:20,585 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:58:20,585 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,585 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:58:20,585 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:58:20,586 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:20,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:20,586 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:20,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:20,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:20,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:20,587 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:20,587 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:58:20,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:20,594 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:58:20,594 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@67118c86[Count = 0] remaining members to acquire global barrier 2023-03-21 11:58:20,594 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:58:20,594 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:20,602 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:20,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:20,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:20,603 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:58:20,603 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:58:20,603 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,603 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:58:20,603 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:58:20,607 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,607 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:58:20,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:20,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:20,607 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:20,607 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:58:20,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:20,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:20,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:20,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:20,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:20,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:58:20,609 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:58:20,609 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:58:20,609 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:58:20,609 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:58:20,609 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:58:20,609 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:58:20,610 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:20,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:20,610 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:20,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:20,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:20,610 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:20,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:20,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:20,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:20,610 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:20,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:20,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:20,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:20,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:20,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:20,611 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:20,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:20,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,613 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,613 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:20,613 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:20,613 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:20,613 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:20,613 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:20,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:20,613 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:58:20,613 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:58:20,613 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:58:20,613 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:20,613 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:20,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:20,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:20,613 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:20,613 DEBUG [Thread-3389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:58:20,613 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:20,613 DEBUG [Thread-3389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:58:20,713 DEBUG [Thread-3389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:58:20,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:58:20,715 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:58:20,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:58:20,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:58:20,716 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:58:20,716 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:20,716 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:58:20,716 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:58:20,716 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:20,716 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:20,717 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,717 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:20,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:20,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:20,717 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,717 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:58:20,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:20,718 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:20,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:58:20,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:20,718 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:58:20,718 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:58:20,718 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:58:20,718 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:58:20,718 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:20,718 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:58:20,718 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:58:20,718 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:58:20,719 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:20,719 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. started... 2023-03-21 11:58:20,719 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. as already flushing 2023-03-21 11:58:20,719 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:20,719 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:58:20,719 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:58:20,719 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,719 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:58:20,719 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:58:20,720 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,720 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:20,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:20,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:20,720 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:20,720 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:58:20,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:20,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:20,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:20,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:20,721 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:58:20,721 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4295b469[Count = 0] remaining members to acquire global barrier 2023-03-21 11:58:20,721 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:58:20,721 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:20,721 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:20,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:20,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:20,722 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:58:20,722 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:58:20,722 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,722 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:58:20,722 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:58:20,722 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,722 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:58:20,722 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:20,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:20,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:20,722 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:58:20,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:20,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:20,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:20,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:20,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:20,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:58:20,724 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:58:20,724 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:58:20,724 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:58:20,724 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:58:20,724 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:58:20,724 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:58:20,725 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:20,725 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:20,725 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:20,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:20,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:20,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:20,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:20,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:20,725 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:20,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:20,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:20,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:20,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:20,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:20,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:20,726 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:20,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:20,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,729 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,729 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:20,729 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:20,729 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:20,729 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:58:20,729 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:58:20,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:20,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:20,729 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:20,729 DEBUG [Thread-3389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:58:20,729 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:20,729 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,729 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:58:20,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:20,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:20,729 DEBUG [Thread-3389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:58:20,729 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:20,730 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:20,730 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:20,794 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:20,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55146 deadline: 1679399960793, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,794 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:20,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55130 deadline: 1679399960793, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,830 DEBUG [Thread-3389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:58:20,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:58:20,831 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:58:20,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:58:20,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:58:20,832 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:58:20,832 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:20,832 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:58:20,832 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:58:20,833 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:20,833 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:20,833 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,833 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:20,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:20,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:20,833 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,834 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:58:20,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:20,834 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:20,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:58:20,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:20,834 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:58:20,834 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:58:20,835 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:58:20,835 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:58:20,835 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:20,835 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:58:20,835 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:58:20,835 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:58:20,835 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:20,835 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. started... 2023-03-21 11:58:20,835 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. as already flushing 2023-03-21 11:58:20,835 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:20,835 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:58:20,835 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:58:20,835 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,835 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:58:20,835 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:58:20,836 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,836 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:20,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:20,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:20,836 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:20,836 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:58:20,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:20,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:20,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:20,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:20,837 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:58:20,837 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7e386129[Count = 0] remaining members to acquire global barrier 2023-03-21 11:58:20,837 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:58:20,837 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:20,838 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:20,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:20,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:20,838 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:58:20,838 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:58:20,838 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:58:20,838 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,838 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:58:20,839 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,839 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:58:20,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:20,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:20,839 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:20,839 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:58:20,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:20,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:20,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:20,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:20,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:20,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:58:20,841 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:58:20,841 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:58:20,841 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:58:20,841 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:58:20,841 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:58:20,841 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:58:20,842 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:20,842 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:20,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:20,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:20,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:20,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:20,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:20,842 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:20,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:20,842 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:20,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:20,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:20,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:20,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:20,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:20,844 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:20,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:20,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,845 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:20,845 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:20,845 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,845 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:58:20,845 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:20,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:20,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:20,845 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:20,845 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:20,846 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,846 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:20,846 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:58:20,846 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:58:20,846 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:20,846 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:20,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:20,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:20,846 DEBUG [Thread-3389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:58:20,846 DEBUG [Thread-3389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:58:20,946 DEBUG [Thread-3389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:58:20,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:58:20,948 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:58:20,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:58:20,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:58:20,949 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:58:20,949 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:20,949 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:58:20,949 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:58:20,950 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:20,950 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:20,950 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:20,950 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:20,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:20,951 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,951 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:58:20,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:20,951 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:20,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:58:20,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:20,951 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:58:20,951 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:58:20,952 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:58:20,952 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:58:20,952 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:20,952 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:58:20,952 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:58:20,952 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:58:20,952 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:20,952 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. started... 2023-03-21 11:58:20,952 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. as already flushing 2023-03-21 11:58:20,952 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:20,952 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:58:20,952 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:58:20,952 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,952 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:58:20,952 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:58:20,953 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,953 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:20,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:20,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:20,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:20,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:20,954 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:20,954 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:58:20,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:20,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:20,955 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:58:20,955 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@25bce233[Count = 0] remaining members to acquire global barrier 2023-03-21 11:58:20,955 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:58:20,955 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:20,955 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:20,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:20,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:20,955 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:58:20,956 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:58:20,956 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:58:20,956 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,956 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:58:20,956 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:58:20,956 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,956 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:20,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:20,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:20,956 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:58:20,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:20,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:20,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:20,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:20,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:20,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:58:20,958 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:58:20,958 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:58:20,958 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:58:20,958 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:58:20,958 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:58:20,958 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:58:20,959 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:20,959 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:20,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:20,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:20,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:20,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:20,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:20,959 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:20,960 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:20,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:20,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:20,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:20,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:20,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:20,960 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:20,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:20,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:20,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,962 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,962 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:20,962 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:20,962 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:20,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:20,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:20,962 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:20,962 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:58:20,962 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:58:20,962 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:20,962 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:20,962 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:20,962 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:20,962 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:20,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:20,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:20,962 DEBUG [Thread-3389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:58:20,962 DEBUG [Thread-3389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:58:20,962 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:58:20,971 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=243, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/A/1c77846499a54017812796e39596f501 2023-03-21 11:58:21,062 DEBUG [Thread-3389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:58:21,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:58:21,064 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:58:21,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:58:21,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:58:21,065 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:58:21,065 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:21,065 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:58:21,065 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:58:21,065 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:21,066 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:21,066 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:21,066 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:21,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:21,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:21,066 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:21,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:21,067 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:58:21,067 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:21,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:58:21,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:21,067 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:58:21,067 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:58:21,067 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:58:21,067 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:58:21,067 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:21,067 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:58:21,067 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:58:21,068 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:58:21,068 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:21,068 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. started... 2023-03-21 11:58:21,068 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. as already flushing 2023-03-21 11:58:21,068 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:21,068 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:58:21,068 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:58:21,068 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:21,068 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:58:21,068 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:58:21,069 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:21,069 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:21,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:21,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:21,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:21,069 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:21,069 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:58:21,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:21,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:21,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:21,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:21,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:21,071 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:58:21,071 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4e223fbe[Count = 0] remaining members to acquire global barrier 2023-03-21 11:58:21,071 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:58:21,071 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:21,071 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:21,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:21,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:21,071 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:58:21,071 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:58:21,071 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:21,071 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:58:21,071 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:58:21,072 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:21,072 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:58:21,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:21,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:21,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:21,072 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:21,072 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:58:21,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:21,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:21,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:21,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:21,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:21,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:21,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:21,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:58:21,074 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:58:21,074 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:58:21,074 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:58:21,074 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:58:21,074 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:58:21,074 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:58:21,075 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:21,075 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:21,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:21,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:21,075 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:21,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:21,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:21,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:21,075 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:21,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:21,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:21,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:21,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:21,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:21,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:21,076 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:21,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:21,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:21,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:21,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:21,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:21,078 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:21,078 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:21,078 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:21,078 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:21,078 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:58:21,078 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:58:21,078 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:21,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:21,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:21,078 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:21,078 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:21,078 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:21,078 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:21,078 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:21,078 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:58:21,078 DEBUG [Thread-3389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:58:21,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:21,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:21,078 DEBUG [Thread-3389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:58:21,178 DEBUG [Thread-3389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:58:21,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:58:21,180 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:58:21,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:58:21,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:58:21,181 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:58:21,181 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:21,181 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:58:21,181 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:58:21,182 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:21,182 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:21,182 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:21,182 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:21,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:21,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:21,182 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:21,183 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:58:21,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:21,183 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:21,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:58:21,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:21,183 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:58:21,183 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:58:21,183 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:58:21,183 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:21,183 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:58:21,183 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:58:21,183 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:58:21,183 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:21,183 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. started... 2023-03-21 11:58:21,183 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. as already flushing 2023-03-21 11:58:21,183 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:21,183 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:58:21,183 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:58:21,184 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:21,184 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:58:21,184 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:58:21,184 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:21,184 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:21,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:21,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:21,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:21,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:21,192 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:21,192 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:58:21,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:21,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:21,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:21,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:21,193 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:58:21,193 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@217bfcef[Count = 0] remaining members to acquire global barrier 2023-03-21 11:58:21,193 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:58:21,193 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:21,198 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:21,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:21,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:21,198 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:58:21,199 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:58:21,199 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:58:21,199 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:21,199 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:58:21,202 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:21,202 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:58:21,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:21,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:21,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:21,202 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:21,203 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:58:21,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:21,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:21,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:21,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:21,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:21,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:21,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:21,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:58:21,204 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:58:21,204 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:58:21,204 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:58:21,204 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:58:21,204 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:58:21,204 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:58:21,205 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:21,205 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:21,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:21,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:21,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:21,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:21,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:21,205 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:21,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:21,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:21,206 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:21,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:21,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:21,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:21,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:21,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:21,207 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:21,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:21,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:21,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:21,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:21,208 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:21,208 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:21,208 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:21,208 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:21,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:21,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:21,208 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:21,208 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:21,208 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:21,208 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:21,208 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:58:21,208 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:58:21,208 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:21,208 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:21,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:21,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:21,209 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:58:21,209 DEBUG [Thread-3389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:58:21,209 DEBUG [Thread-3389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:58:21,299 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:21,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55146 deadline: 1679399961297, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:21,301 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:21,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55130 deadline: 1679399961299, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:21,309 DEBUG [Thread-3389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:58:21,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:58:21,310 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:58:21,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:58:21,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:58:21,311 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:58:21,312 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:21,312 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:58:21,312 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:58:21,312 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:21,312 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:21,312 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:21,312 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:21,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:21,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:21,313 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:21,313 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:58:21,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:21,313 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:21,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:58:21,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:21,313 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:58:21,314 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:58:21,314 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:58:21,314 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:58:21,314 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:21,314 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:58:21,314 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:58:21,314 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:58:21,314 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:21,314 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. started... 2023-03-21 11:58:21,314 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. as already flushing 2023-03-21 11:58:21,314 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:21,314 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:58:21,314 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:58:21,314 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:21,314 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:58:21,314 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:58:21,315 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:21,315 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:21,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:21,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:21,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:21,315 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:21,315 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:58:21,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:21,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:21,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:21,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:21,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:21,316 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:58:21,316 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c77e0c0[Count = 0] remaining members to acquire global barrier 2023-03-21 11:58:21,316 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:58:21,316 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:21,317 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:21,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:21,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:21,317 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:58:21,317 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:58:21,317 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:21,317 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:58:21,317 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:58:21,318 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:21,318 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:58:21,318 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:21,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:21,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:21,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:21,318 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:58:21,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:21,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:21,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:21,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:21,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:21,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:21,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:21,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:58:21,320 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:58:21,320 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:58:21,320 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:58:21,320 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:58:21,320 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:58:21,320 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:58:21,321 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:21,321 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:21,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:21,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:21,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:21,321 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:21,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:21,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:21,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:21,321 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:21,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:21,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:21,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:21,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:21,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:21,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:21,322 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:21,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:21,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:21,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:21,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:21,323 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:21,323 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:21,323 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:21,323 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:21,323 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:21,323 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:58:21,323 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:58:21,324 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:58:21,323 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:21,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:21,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:21,323 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:21,324 DEBUG [Thread-3389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:58:21,324 DEBUG [Thread-3389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:58:21,324 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:21,324 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:21,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:21,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:21,324 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:21,379 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=243 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/B/bd6064f55aa54d9390d7b514465cf766 2023-03-21 11:58:21,385 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=243 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/C/b948beeaa4184106bfe62a6aaee49c51 2023-03-21 11:58:21,388 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/A/1c77846499a54017812796e39596f501 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/1c77846499a54017812796e39596f501 2023-03-21 11:58:21,390 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/1c77846499a54017812796e39596f501, entries=200, sequenceid=243, filesize=32.1 K 2023-03-21 11:58:21,391 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/B/bd6064f55aa54d9390d7b514465cf766 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/bd6064f55aa54d9390d7b514465cf766 2023-03-21 11:58:21,393 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/bd6064f55aa54d9390d7b514465cf766, entries=150, sequenceid=243, filesize=11.8 K 2023-03-21 11:58:21,394 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/C/b948beeaa4184106bfe62a6aaee49c51 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/b948beeaa4184106bfe62a6aaee49c51 2023-03-21 11:58:21,396 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/b948beeaa4184106bfe62a6aaee49c51, entries=150, sequenceid=243, filesize=11.8 K 2023-03-21 11:58:21,397 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for cc711a8c4e0d80e60aabbf61e79ab961 in 1252ms, sequenceid=243, compaction requested=true 2023-03-21 11:58:21,397 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for cc711a8c4e0d80e60aabbf61e79ab961: 2023-03-21 11:58:21,397 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-21 11:58:21,397 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-03-21 11:58:21,397 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-03-21 11:58:21,397 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-21 11:58:21,397 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-21 11:58:21,399 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 84342 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-03-21 11:58:21,399 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 188352 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-03-21 11:58:21,399 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1912): cc711a8c4e0d80e60aabbf61e79ab961/B is initiating minor compaction (all files) 2023-03-21 11:58:21,399 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1912): cc711a8c4e0d80e60aabbf61e79ab961/A is initiating minor compaction (all files) 2023-03-21 11:58:21,399 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegion(2259): Starting compaction of cc711a8c4e0d80e60aabbf61e79ab961/B in TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:21,399 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of cc711a8c4e0d80e60aabbf61e79ab961/A in TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:21,399 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/c7cb4612afbc4523b757d4c69a823c1b, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/3e379f97af55485191c99a71435d9a11, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/e8b79d7e121f4c82b00768a168e3334f, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/8918745ddf7e412d986b8d3b68703819, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/c6c884a4724648c88009f1b520daa5a1, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/af95ed8c42ca478cb3e7dce00bc46237, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/bd6064f55aa54d9390d7b514465cf766] into tmpdir=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp, totalSize=82.4 K 2023-03-21 11:58:21,399 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/f2c16c492d9046a396017eae634bbec2, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/a98a48d8082a4ca1b951afa24ad40869, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/b3bcf43bd78440f882303558ec4be526, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/defb923ec42b470cb29d18cd54b28bd4, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/292986e6656c4c3693e3a2f2f600a7d2, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/5c23338f1a8240e08d210963d0f06716, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/1c77846499a54017812796e39596f501] into tmpdir=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp, totalSize=183.9 K 2023-03-21 11:58:21,400 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting c7cb4612afbc4523b757d4c69a823c1b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=99, earliestPutTs=1679399893341 2023-03-21 11:58:21,400 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting f2c16c492d9046a396017eae634bbec2, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=99, earliestPutTs=1679399893341 2023-03-21 11:58:21,400 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 3e379f97af55485191c99a71435d9a11, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=134, earliestPutTs=1679399893390 2023-03-21 11:58:21,400 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting a98a48d8082a4ca1b951afa24ad40869, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=134, earliestPutTs=1679399893390 2023-03-21 11:58:21,400 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting e8b79d7e121f4c82b00768a168e3334f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=148, earliestPutTs=1679399895504 2023-03-21 11:58:21,400 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting b3bcf43bd78440f882303558ec4be526, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=148, earliestPutTs=1679399895504 2023-03-21 11:58:21,400 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 8918745ddf7e412d986b8d3b68703819, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=184, earliestPutTs=1679399896173 2023-03-21 11:58:21,400 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting defb923ec42b470cb29d18cd54b28bd4, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=184, earliestPutTs=1679399896173 2023-03-21 11:58:21,400 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting c6c884a4724648c88009f1b520daa5a1, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=195, earliestPutTs=1679399897261 2023-03-21 11:58:21,401 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 292986e6656c4c3693e3a2f2f600a7d2, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=195, earliestPutTs=1679399897261 2023-03-21 11:58:21,401 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting af95ed8c42ca478cb3e7dce00bc46237, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=231, earliestPutTs=1679399898398 2023-03-21 11:58:21,401 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 5c23338f1a8240e08d210963d0f06716, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=231, earliestPutTs=1679399898398 2023-03-21 11:58:21,401 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting bd6064f55aa54d9390d7b514465cf766, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=243, earliestPutTs=1679399899527 2023-03-21 11:58:21,401 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 1c77846499a54017812796e39596f501, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=243, earliestPutTs=1679399899527 2023-03-21 11:58:21,410 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] throttle.PressureAwareThroughputController(145): cc711a8c4e0d80e60aabbf61e79ab961#B#compaction#474 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 57.69 MB/second 2023-03-21 11:58:21,414 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] throttle.PressureAwareThroughputController(145): cc711a8c4e0d80e60aabbf61e79ab961#A#compaction#475 average throughput is 2.82 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 57.69 MB/second 2023-03-21 11:58:21,424 DEBUG [Thread-3389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:58:21,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:58:21,426 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:58:21,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:58:21,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:58:21,427 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:58:21,427 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:21,427 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:58:21,427 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:58:21,427 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:21,427 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:21,428 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:21,428 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:21,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:21,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:21,428 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:21,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:21,428 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:58:21,428 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:21,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:58:21,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:21,428 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:58:21,429 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:58:21,429 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:58:21,429 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:58:21,429 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:21,429 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:58:21,429 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:58:21,429 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:58:21,429 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:21,429 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. started... 2023-03-21 11:58:21,429 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing cc711a8c4e0d80e60aabbf61e79ab961 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-21 11:58:21,429 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cc711a8c4e0d80e60aabbf61e79ab961, store=A 2023-03-21 11:58:21,429 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:58:21,429 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cc711a8c4e0d80e60aabbf61e79ab961, store=B 2023-03-21 11:58:21,430 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:58:21,430 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cc711a8c4e0d80e60aabbf61e79ab961, store=C 2023-03-21 11:58:21,430 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:58:21,443 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303211b4330e84fa84291bb453d5b123a0403 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303211b4330e84fa84291bb453d5b123a0403 2023-03-21 11:58:21,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(9158): Flush requested on cc711a8c4e0d80e60aabbf61e79ab961 2023-03-21 11:58:21,565 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. as already flushing 2023-03-21 11:58:21,576 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:21,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55174 deadline: 1679399961573, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:21,579 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:21,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55112 deadline: 1679399961576, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:21,579 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:21,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55178 deadline: 1679399961576, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:21,679 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:21,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55174 deadline: 1679399961677, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:21,682 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:21,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55112 deadline: 1679399961680, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:21,683 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:21,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55178 deadline: 1679399961680, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:21,817 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/B/078103d4734849d4a11a964db01a0135 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/078103d4734849d4a11a964db01a0135 2023-03-21 11:58:21,821 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in cc711a8c4e0d80e60aabbf61e79ab961/B of cc711a8c4e0d80e60aabbf61e79ab961 into 078103d4734849d4a11a964db01a0135(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-21 11:58:21,821 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for cc711a8c4e0d80e60aabbf61e79ab961: 2023-03-21 11:58:21,821 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961., storeName=cc711a8c4e0d80e60aabbf61e79ab961/B, priority=9, startTime=1679399901397; duration=0sec 2023-03-21 11:58:21,821 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-21 11:58:21,821 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-03-21 11:58:21,823 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 84342 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-03-21 11:58:21,823 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1912): cc711a8c4e0d80e60aabbf61e79ab961/C is initiating minor compaction (all files) 2023-03-21 11:58:21,823 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegion(2259): Starting compaction of cc711a8c4e0d80e60aabbf61e79ab961/C in TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:21,823 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/9e5ec838908846808d55649fb1447353, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/f9fd1189769a4e55a7b416911e3bb8a3, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/260e444f57bc496eaa872ac7cfaaf58d, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/57d09b7c6c164e71b25efd5aa897f8f1, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/e46a05e063434332b9847c59242eba44, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/47ffd81cd1cf481d865ab08a2d1330b9, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/b948beeaa4184106bfe62a6aaee49c51] into tmpdir=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp, totalSize=82.4 K 2023-03-21 11:58:21,823 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 9e5ec838908846808d55649fb1447353, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=99, earliestPutTs=1679399893341 2023-03-21 11:58:21,824 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting f9fd1189769a4e55a7b416911e3bb8a3, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=134, earliestPutTs=1679399893390 2023-03-21 11:58:21,826 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 260e444f57bc496eaa872ac7cfaaf58d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=148, earliestPutTs=1679399895504 2023-03-21 11:58:21,827 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 57d09b7c6c164e71b25efd5aa897f8f1, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=184, earliestPutTs=1679399896173 2023-03-21 11:58:21,827 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting e46a05e063434332b9847c59242eba44, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=195, earliestPutTs=1679399897261 2023-03-21 11:58:21,827 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 47ffd81cd1cf481d865ab08a2d1330b9, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=231, earliestPutTs=1679399898398 2023-03-21 11:58:21,827 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting b948beeaa4184106bfe62a6aaee49c51, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=243, earliestPutTs=1679399899527 2023-03-21 11:58:21,830 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/A/f0dcbc97bde34471a9a13e7be84e8055 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/f0dcbc97bde34471a9a13e7be84e8055 2023-03-21 11:58:21,833 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in cc711a8c4e0d80e60aabbf61e79ab961/A of cc711a8c4e0d80e60aabbf61e79ab961 into f0dcbc97bde34471a9a13e7be84e8055(size=25.8 K), total size for store is 25.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-21 11:58:21,833 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for cc711a8c4e0d80e60aabbf61e79ab961: 2023-03-21 11:58:21,833 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961., storeName=cc711a8c4e0d80e60aabbf61e79ab961/A, priority=9, startTime=1679399901397; duration=0sec 2023-03-21 11:58:21,833 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-21 11:58:21,837 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] throttle.PressureAwareThroughputController(145): cc711a8c4e0d80e60aabbf61e79ab961#C#compaction#477 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 57.69 MB/second 2023-03-21 11:58:21,847 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=263, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/A/ccb54acca49a438c84dc55e0f6bc5e6d 2023-03-21 11:58:21,854 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=263 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/B/f69bacda2a984ccba2ac2e4dbd536e1d 2023-03-21 11:58:21,883 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:21,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55174 deadline: 1679399961881, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:21,886 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:21,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55112 deadline: 1679399961883, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:21,887 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:21,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55178 deadline: 1679399961884, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:22,187 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:22,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55174 deadline: 1679399962186, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:22,189 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:22,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55112 deadline: 1679399962188, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:22,191 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:22,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55178 deadline: 1679399962189, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:22,247 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/C/7739b5dc3de04798a076d8be162b3aea as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/7739b5dc3de04798a076d8be162b3aea 2023-03-21 11:58:22,254 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in cc711a8c4e0d80e60aabbf61e79ab961/C of cc711a8c4e0d80e60aabbf61e79ab961 into 7739b5dc3de04798a076d8be162b3aea(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-21 11:58:22,254 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for cc711a8c4e0d80e60aabbf61e79ab961: 2023-03-21 11:58:22,254 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961., storeName=cc711a8c4e0d80e60aabbf61e79ab961/C, priority=9, startTime=1679399901397; duration=0sec 2023-03-21 11:58:22,254 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-21 11:58:22,262 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=263 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/C/9f44ee270f3a4a4c95fe8792635a3688 2023-03-21 11:58:22,266 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/A/ccb54acca49a438c84dc55e0f6bc5e6d as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/ccb54acca49a438c84dc55e0f6bc5e6d 2023-03-21 11:58:22,269 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/ccb54acca49a438c84dc55e0f6bc5e6d, entries=150, sequenceid=263, filesize=25.4 K 2023-03-21 11:58:22,270 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/B/f69bacda2a984ccba2ac2e4dbd536e1d as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/f69bacda2a984ccba2ac2e4dbd536e1d 2023-03-21 11:58:22,273 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/f69bacda2a984ccba2ac2e4dbd536e1d, entries=150, sequenceid=263, filesize=11.9 K 2023-03-21 11:58:22,273 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/C/9f44ee270f3a4a4c95fe8792635a3688 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/9f44ee270f3a4a4c95fe8792635a3688 2023-03-21 11:58:22,276 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/9f44ee270f3a4a4c95fe8792635a3688, entries=150, sequenceid=263, filesize=11.9 K 2023-03-21 11:58:22,277 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=67.09 KB/68700 for cc711a8c4e0d80e60aabbf61e79ab961 in 848ms, sequenceid=263, compaction requested=false 2023-03-21 11:58:22,277 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for cc711a8c4e0d80e60aabbf61e79ab961: 2023-03-21 11:58:22,277 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:22,277 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:58:22,277 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:58:22,277 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:22,277 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:58:22,277 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:58:22,278 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:22,278 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:22,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:22,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:22,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:22,278 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:22,278 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:58:22,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:22,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:22,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:22,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:22,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:22,279 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:58:22,279 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e983407[Count = 0] remaining members to acquire global barrier 2023-03-21 11:58:22,279 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:58:22,280 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:22,281 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:22,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:22,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:22,281 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:58:22,281 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:58:22,281 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:58:22,281 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:22,281 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:58:22,282 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:58:22,282 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:22,282 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:22,282 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:58:22,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:22,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:22,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:22,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:22,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:22,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:22,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:22,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:22,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:22,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:22,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:58:22,284 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:58:22,284 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:58:22,284 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:58:22,284 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:58:22,284 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:58:22,284 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:58:22,285 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:22,285 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:22,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:22,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:22,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:22,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:22,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:22,285 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:22,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:22,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:22,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:22,286 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:22,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:22,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:22,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:22,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:22,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:22,287 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:22,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:22,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:22,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:22,289 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:22,289 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:22,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:22,289 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:22,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:22,289 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:58:22,289 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:58:22,289 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:22,290 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:22,290 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:22,290 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:22,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:22,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:22,290 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:22,290 DEBUG [Thread-3389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:58:22,290 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:22,290 DEBUG [Thread-3389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:58:22,290 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:22,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(9158): Flush requested on cc711a8c4e0d80e60aabbf61e79ab961 2023-03-21 11:58:22,302 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing cc711a8c4e0d80e60aabbf61e79ab961 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-03-21 11:58:22,302 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cc711a8c4e0d80e60aabbf61e79ab961, store=A 2023-03-21 11:58:22,302 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:58:22,302 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cc711a8c4e0d80e60aabbf61e79ab961, store=B 2023-03-21 11:58:22,302 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:58:22,302 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cc711a8c4e0d80e60aabbf61e79ab961, store=C 2023-03-21 11:58:22,302 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:58:22,318 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230321980f5c523f9c439a9c1ecf97405b115e to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230321980f5c523f9c439a9c1ecf97405b115e 2023-03-21 11:58:22,323 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=280, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/A/28c08eb034584514ba21d92b7100b425 2023-03-21 11:58:22,335 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:22,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55130 deadline: 1679399962333, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:22,335 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:22,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55146 deadline: 1679399962333, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:22,390 DEBUG [Thread-3389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:58:22,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:58:22,392 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:58:22,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:58:22,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:58:22,393 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:58:22,393 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:22,393 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:58:22,393 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:58:22,393 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:22,393 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:22,394 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:22,394 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:22,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:22,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:22,394 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:22,394 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:58:22,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:22,394 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:22,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:58:22,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:22,395 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:58:22,395 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:58:22,395 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:58:22,395 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:58:22,395 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:22,395 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:58:22,395 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:58:22,395 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:58:22,395 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:22,395 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. started... 2023-03-21 11:58:22,395 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. as already flushing 2023-03-21 11:58:22,395 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:22,395 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:58:22,395 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:58:22,396 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:22,396 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:58:22,396 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:58:22,396 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:22,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:22,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:22,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:22,396 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:22,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:22,397 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:22,397 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:58:22,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:22,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:22,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:22,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:22,399 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:58:22,399 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1182fa5[Count = 0] remaining members to acquire global barrier 2023-03-21 11:58:22,399 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:58:22,399 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:22,399 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:22,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:22,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:22,400 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:58:22,400 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:58:22,400 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:58:22,400 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:22,400 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:58:22,401 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:22,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:22,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:22,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:22,401 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:58:22,401 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:22,401 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:58:22,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:22,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:22,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:22,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:22,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:22,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:22,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:22,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:58:22,402 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:58:22,402 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:58:22,402 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:58:22,402 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:58:22,402 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:58:22,403 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:58:22,403 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:22,403 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:22,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:22,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:22,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:22,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:22,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:22,403 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:22,404 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:22,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:22,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:22,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:22,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:22,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:22,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:22,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:22,405 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:22,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:22,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:22,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:22,406 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:22,406 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:22,406 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:22,406 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:22,406 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:22,406 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:22,406 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:58:22,406 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:58:22,406 DEBUG [Thread-3389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:58:22,406 DEBUG [Thread-3389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:58:22,406 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:22,406 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:58:22,406 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:22,406 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:22,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:22,407 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:22,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:22,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:22,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:22,438 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:22,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55130 deadline: 1679399962436, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:22,438 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:22,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55146 deadline: 1679399962436, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:22,506 DEBUG [Thread-3389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:58:22,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:58:22,508 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:58:22,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:58:22,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:58:22,509 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:58:22,509 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:22,509 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:58:22,509 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:58:22,510 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:22,510 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:22,510 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:22,510 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:22,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:22,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:22,510 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:22,510 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:58:22,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:22,511 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:22,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:58:22,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:22,511 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:58:22,511 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:58:22,511 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:58:22,511 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:58:22,511 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:22,511 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:58:22,511 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:58:22,512 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:58:22,512 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:22,512 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. started... 2023-03-21 11:58:22,512 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. as already flushing 2023-03-21 11:58:22,512 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:22,512 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:58:22,512 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:58:22,512 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:22,512 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:58:22,512 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:58:22,513 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:22,513 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:22,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:22,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:22,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:22,513 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:22,513 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:58:22,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:22,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:22,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:22,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:22,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:22,514 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:58:22,514 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f7a37ed[Count = 0] remaining members to acquire global barrier 2023-03-21 11:58:22,514 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:58:22,514 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:22,515 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:22,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:22,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:22,515 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:58:22,515 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:58:22,515 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:58:22,515 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:22,515 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:58:22,515 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:58:22,515 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:22,515 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:22,515 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:58:22,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:22,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:22,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:22,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:22,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:22,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:22,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:22,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:22,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:22,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:22,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:58:22,517 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:58:22,517 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:58:22,518 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:58:22,518 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:58:22,518 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:58:22,518 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:58:22,519 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:22,519 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:22,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:22,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:22,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:22,519 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:22,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:22,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:22,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:22,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:22,519 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:22,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:22,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:22,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:22,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:22,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:22,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:22,520 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:22,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:22,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:22,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:22,522 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:22,522 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:22,522 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:22,522 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:22,522 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:22,522 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:58:22,522 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:58:22,522 DEBUG [Thread-3389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:58:22,523 DEBUG [Thread-3389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:58:22,522 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:22,522 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:22,522 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:22,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:22,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:22,523 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:22,523 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:22,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:22,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:22,623 DEBUG [Thread-3389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:58:22,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:58:22,624 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:58:22,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:58:22,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:58:22,625 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:58:22,626 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:22,626 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:58:22,626 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:58:22,626 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:22,626 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:22,627 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:22,627 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:22,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:22,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:22,627 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:22,627 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:58:22,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:22,627 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:22,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:58:22,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:22,627 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:58:22,630 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:58:22,630 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:58:22,630 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:22,630 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:58:22,630 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:58:22,630 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:58:22,630 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:22,630 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. started... 2023-03-21 11:58:22,630 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. as already flushing 2023-03-21 11:58:22,630 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:22,630 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:58:22,630 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:58:22,630 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:22,630 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:58:22,630 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:58:22,631 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:22,631 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:22,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:22,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:22,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:22,631 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:22,631 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:58:22,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:22,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:22,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:22,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:22,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:22,632 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:58:22,633 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ca8b922[Count = 0] remaining members to acquire global barrier 2023-03-21 11:58:22,633 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:58:22,633 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:22,634 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:22,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:22,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:22,634 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:58:22,634 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:58:22,634 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:58:22,634 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:22,634 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:58:22,635 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:22,635 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:58:22,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:22,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:22,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:22,635 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:22,635 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:58:22,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:22,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:22,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:22,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:22,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:22,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:22,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:22,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:58:22,637 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:58:22,637 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:58:22,637 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:58:22,637 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:58:22,637 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:58:22,637 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:58:22,639 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:22,639 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:22,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:22,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:22,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:22,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:22,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:22,639 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:22,639 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:22,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:22,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:22,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:22,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:22,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:22,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:22,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:22,641 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:22,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:22,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:22,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:22,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:22,642 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:22,642 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:22,642 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:22,642 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:22,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:22,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:22,642 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:22,642 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:58:22,642 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:58:22,642 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:22,643 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:58:22,642 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:22,643 DEBUG [Thread-3389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:58:22,643 DEBUG [Thread-3389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:58:22,642 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:22,642 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:22,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:22,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:22,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55146 deadline: 1679399962640, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:22,643 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:22,643 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:22,643 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:22,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55130 deadline: 1679399962641, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:22,692 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:22,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55174 deadline: 1679399962691, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:22,692 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:22,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55178 deadline: 1679399962691, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:22,693 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:22,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55112 deadline: 1679399962691, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:22,730 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=280 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/B/4be497b2982d4a11b692214031e223f1 2023-03-21 11:58:22,737 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=280 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/C/895fb44174e64a1ba10b6979dd319430 2023-03-21 11:58:22,740 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/A/28c08eb034584514ba21d92b7100b425 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/28c08eb034584514ba21d92b7100b425 2023-03-21 11:58:22,743 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/28c08eb034584514ba21d92b7100b425, entries=200, sequenceid=280, filesize=32.3 K 2023-03-21 11:58:22,743 DEBUG [Thread-3389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:58:22,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:58:22,744 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:58:22,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:58:22,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:58:22,745 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:58:22,745 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:22,745 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:58:22,745 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:58:22,745 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:22,745 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:22,746 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/B/4be497b2982d4a11b692214031e223f1 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/4be497b2982d4a11b692214031e223f1 2023-03-21 11:58:22,746 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:22,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:22,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:22,746 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:22,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:22,747 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:22,747 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:58:22,747 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:22,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:58:22,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:22,747 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:58:22,747 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:58:22,747 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:58:22,747 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:58:22,747 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:22,748 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:58:22,748 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:58:22,748 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:22,748 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:58:22,748 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. started... 2023-03-21 11:58:22,748 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. as already flushing 2023-03-21 11:58:22,748 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:22,748 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:58:22,748 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:58:22,748 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:22,748 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:58:22,748 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:58:22,749 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:22,749 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:22,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:22,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:22,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:22,749 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:22,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:22,749 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:58:22,749 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/4be497b2982d4a11b692214031e223f1, entries=150, sequenceid=280, filesize=11.9 K 2023-03-21 11:58:22,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:22,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:22,750 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/C/895fb44174e64a1ba10b6979dd319430 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/895fb44174e64a1ba10b6979dd319430 2023-03-21 11:58:22,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:22,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:22,751 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:58:22,751 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@430f86bc[Count = 0] remaining members to acquire global barrier 2023-03-21 11:58:22,751 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:58:22,751 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:22,751 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:22,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:22,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:22,752 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:58:22,752 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:58:22,752 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:58:22,752 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:22,752 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:58:22,752 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:22,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:22,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:22,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:22,752 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:58:22,753 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:22,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:22,753 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:58:22,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:22,753 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/895fb44174e64a1ba10b6979dd319430, entries=150, sequenceid=280, filesize=11.9 K 2023-03-21 11:58:22,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:22,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:22,754 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for cc711a8c4e0d80e60aabbf61e79ab961 in 452ms, sequenceid=280, compaction requested=true 2023-03-21 11:58:22,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:22,754 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for cc711a8c4e0d80e60aabbf61e79ab961: 2023-03-21 11:58:22,754 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-21 11:58:22,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:22,754 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-21 11:58:22,754 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-21 11:58:22,754 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-21 11:58:22,754 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-21 11:58:22,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:22,755 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 85506 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-21 11:58:22,755 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1912): cc711a8c4e0d80e60aabbf61e79ab961/A is initiating minor compaction (all files) 2023-03-21 11:58:22,755 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of cc711a8c4e0d80e60aabbf61e79ab961/A in TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:22,755 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/f0dcbc97bde34471a9a13e7be84e8055, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/ccb54acca49a438c84dc55e0f6bc5e6d, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/28c08eb034584514ba21d92b7100b425] into tmpdir=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp, totalSize=83.5 K 2023-03-21 11:58:22,755 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36886 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-21 11:58:22,755 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1912): cc711a8c4e0d80e60aabbf61e79ab961/B is initiating minor compaction (all files) 2023-03-21 11:58:22,755 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegion(2259): Starting compaction of cc711a8c4e0d80e60aabbf61e79ab961/B in TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:22,755 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/078103d4734849d4a11a964db01a0135, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/f69bacda2a984ccba2ac2e4dbd536e1d, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/4be497b2982d4a11b692214031e223f1] into tmpdir=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp, totalSize=36.0 K 2023-03-21 11:58:22,755 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting f0dcbc97bde34471a9a13e7be84e8055, keycount=150, bloomtype=ROW, size=25.8 K, encoding=NONE, compression=NONE, seqNum=243, earliestPutTs=1679399899527 2023-03-21 11:58:22,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:58:22,756 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:58:22,756 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:58:22,756 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:58:22,756 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:58:22,756 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:58:22,756 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:58:22,756 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 078103d4734849d4a11a964db01a0135, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=243, earliestPutTs=1679399899527 2023-03-21 11:58:22,756 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting ccb54acca49a438c84dc55e0f6bc5e6d, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=263, earliestPutTs=1679399900173 2023-03-21 11:58:22,756 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting f69bacda2a984ccba2ac2e4dbd536e1d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=263, earliestPutTs=1679399900173 2023-03-21 11:58:22,756 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 28c08eb034584514ba21d92b7100b425, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=280, earliestPutTs=1679399901572 2023-03-21 11:58:22,757 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 4be497b2982d4a11b692214031e223f1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=280, earliestPutTs=1679399901572 2023-03-21 11:58:22,757 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:22,757 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:22,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:22,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:22,757 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:22,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:22,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:22,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:22,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:22,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:22,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:22,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:22,758 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:22,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:22,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:22,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:22,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:22,760 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:22,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:22,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:22,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:22,761 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:22,761 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:22,761 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:58:22,762 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:58:22,761 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:22,762 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:22,762 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:22,762 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:22,762 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:22,762 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:22,761 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:22,761 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:22,763 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] throttle.PressureAwareThroughputController(145): cc711a8c4e0d80e60aabbf61e79ab961#B#compaction#483 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 57.69 MB/second 2023-03-21 11:58:22,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:22,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:22,762 DEBUG [Thread-3389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:58:22,764 DEBUG [Thread-3389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:58:22,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:22,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:22,769 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] throttle.PressureAwareThroughputController(145): cc711a8c4e0d80e60aabbf61e79ab961#A#compaction#484 average throughput is 3.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 57.69 MB/second 2023-03-21 11:58:22,771 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/B/c3ba93acec384b11a8b1c70491964730 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/c3ba93acec384b11a8b1c70491964730 2023-03-21 11:58:22,775 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in cc711a8c4e0d80e60aabbf61e79ab961/B of cc711a8c4e0d80e60aabbf61e79ab961 into c3ba93acec384b11a8b1c70491964730(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-21 11:58:22,775 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for cc711a8c4e0d80e60aabbf61e79ab961: 2023-03-21 11:58:22,775 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961., storeName=cc711a8c4e0d80e60aabbf61e79ab961/B, priority=13, startTime=1679399902754; duration=0sec 2023-03-21 11:58:22,775 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-21 11:58:22,775 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-21 11:58:22,776 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36886 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-21 11:58:22,776 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1912): cc711a8c4e0d80e60aabbf61e79ab961/C is initiating minor compaction (all files) 2023-03-21 11:58:22,776 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegion(2259): Starting compaction of cc711a8c4e0d80e60aabbf61e79ab961/C in TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:22,776 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/7739b5dc3de04798a076d8be162b3aea, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/9f44ee270f3a4a4c95fe8792635a3688, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/895fb44174e64a1ba10b6979dd319430] into tmpdir=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp, totalSize=36.0 K 2023-03-21 11:58:22,777 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 7739b5dc3de04798a076d8be162b3aea, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=243, earliestPutTs=1679399899527 2023-03-21 11:58:22,777 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 9f44ee270f3a4a4c95fe8792635a3688, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=263, earliestPutTs=1679399900173 2023-03-21 11:58:22,777 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 895fb44174e64a1ba10b6979dd319430, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=280, earliestPutTs=1679399901572 2023-03-21 11:58:22,783 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] throttle.PressureAwareThroughputController(145): cc711a8c4e0d80e60aabbf61e79ab961#C#compaction#485 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 57.69 MB/second 2023-03-21 11:58:22,790 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/C/16ad84427f934446b251aefeab82a0aa as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/16ad84427f934446b251aefeab82a0aa 2023-03-21 11:58:22,793 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in cc711a8c4e0d80e60aabbf61e79ab961/C of cc711a8c4e0d80e60aabbf61e79ab961 into 16ad84427f934446b251aefeab82a0aa(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-21 11:58:22,793 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for cc711a8c4e0d80e60aabbf61e79ab961: 2023-03-21 11:58:22,793 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961., storeName=cc711a8c4e0d80e60aabbf61e79ab961/C, priority=13, startTime=1679399902754; duration=0sec 2023-03-21 11:58:22,793 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-21 11:58:22,864 DEBUG [Thread-3389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:58:22,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:58:22,865 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:58:22,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:58:22,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:58:22,866 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:58:22,866 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:22,866 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:58:22,866 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:58:22,867 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:22,867 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:22,867 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:22,867 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:22,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:22,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:22,868 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:22,868 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:58:22,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:22,868 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:22,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:58:22,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:22,868 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:58:22,868 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:58:22,868 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:58:22,869 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:58:22,869 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:22,869 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:58:22,869 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:58:22,869 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:58:22,869 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:22,869 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. started... 2023-03-21 11:58:22,869 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing cc711a8c4e0d80e60aabbf61e79ab961 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-03-21 11:58:22,869 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cc711a8c4e0d80e60aabbf61e79ab961, store=A 2023-03-21 11:58:22,869 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:58:22,869 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cc711a8c4e0d80e60aabbf61e79ab961, store=B 2023-03-21 11:58:22,869 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:58:22,869 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cc711a8c4e0d80e60aabbf61e79ab961, store=C 2023-03-21 11:58:22,869 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:58:22,877 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303213a2bed14280543148d0ef360c4a5a260 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303213a2bed14280543148d0ef360c4a5a260 2023-03-21 11:58:22,885 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=300, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/A/2020c8cf2586406f8b2bf6bd2b1c6859 2023-03-21 11:58:22,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(9158): Flush requested on cc711a8c4e0d80e60aabbf61e79ab961 2023-03-21 11:58:22,946 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. as already flushing 2023-03-21 11:58:22,960 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:22,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55130 deadline: 1679399962959, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:22,961 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:22,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55146 deadline: 1679399962959, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:23,063 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:23,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55130 deadline: 1679399963061, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:23,063 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:23,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55146 deadline: 1679399963061, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:23,265 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:23,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55130 deadline: 1679399963264, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:23,265 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:23,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55146 deadline: 1679399963265, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:23,296 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=300 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/B/fb4972bf9dc445fa8d5a01e581753210 2023-03-21 11:58:23,304 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=300 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/C/712739b02b2b4d918fea140377c092fd 2023-03-21 11:58:23,307 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/A/2020c8cf2586406f8b2bf6bd2b1c6859 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/2020c8cf2586406f8b2bf6bd2b1c6859 2023-03-21 11:58:23,309 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/2020c8cf2586406f8b2bf6bd2b1c6859, entries=150, sequenceid=300, filesize=25.4 K 2023-03-21 11:58:23,310 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/B/fb4972bf9dc445fa8d5a01e581753210 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/fb4972bf9dc445fa8d5a01e581753210 2023-03-21 11:58:23,313 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/fb4972bf9dc445fa8d5a01e581753210, entries=150, sequenceid=300, filesize=11.9 K 2023-03-21 11:58:23,313 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/C/712739b02b2b4d918fea140377c092fd as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/712739b02b2b4d918fea140377c092fd 2023-03-21 11:58:23,316 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/712739b02b2b4d918fea140377c092fd, entries=150, sequenceid=300, filesize=11.9 K 2023-03-21 11:58:23,316 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for cc711a8c4e0d80e60aabbf61e79ab961 in 447ms, sequenceid=300, compaction requested=false 2023-03-21 11:58:23,316 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for cc711a8c4e0d80e60aabbf61e79ab961: 2023-03-21 11:58:23,316 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:23,317 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:58:23,317 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:58:23,317 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:23,317 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:58:23,317 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:58:23,317 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:23,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:23,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:23,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:23,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:23,318 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:23,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:23,318 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:23,318 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:58:23,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:23,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:23,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:23,319 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:58:23,319 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3aafc7f[Count = 0] remaining members to acquire global barrier 2023-03-21 11:58:23,319 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:58:23,319 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:23,320 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:23,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:23,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:23,320 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:58:23,320 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:58:23,320 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:58:23,320 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:23,320 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:58:23,321 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:23,321 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:58:23,321 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:23,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:23,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:23,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:23,321 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:58:23,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:23,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:23,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:23,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:23,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:23,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:23,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:23,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:58:23,323 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:58:23,323 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:58:23,323 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:58:23,323 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:58:23,323 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:58:23,323 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:58:23,324 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:23,324 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:23,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:23,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:23,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:23,324 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:23,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:23,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:23,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:23,325 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:23,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:23,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:23,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:23,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:23,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:23,326 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:23,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:23,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:23,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:23,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:23,327 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:23,327 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:23,327 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:23,327 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:23,327 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:58:23,327 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:58:23,327 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:58:23,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:23,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:23,327 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:23,327 DEBUG [Thread-3389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:58:23,328 DEBUG [Thread-3389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:58:23,327 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:23,327 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:23,328 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:23,328 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:23,328 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:23,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:23,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:23,428 DEBUG [Thread-3389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:58:23,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:58:23,429 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:58:23,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:58:23,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:58:23,430 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:58:23,430 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:23,430 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:58:23,430 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:58:23,431 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:23,431 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:23,431 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:23,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:23,431 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:23,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:23,432 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:23,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:23,432 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:58:23,432 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:23,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:58:23,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:23,432 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:58:23,432 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:58:23,432 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:58:23,432 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:58:23,432 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:23,432 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:58:23,432 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:58:23,432 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:23,432 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. started... 2023-03-21 11:58:23,432 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:58:23,433 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing cc711a8c4e0d80e60aabbf61e79ab961 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-03-21 11:58:23,433 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cc711a8c4e0d80e60aabbf61e79ab961, store=A 2023-03-21 11:58:23,433 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:58:23,433 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cc711a8c4e0d80e60aabbf61e79ab961, store=B 2023-03-21 11:58:23,433 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:58:23,433 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cc711a8c4e0d80e60aabbf61e79ab961, store=C 2023-03-21 11:58:23,433 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:58:23,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(9158): Flush requested on cc711a8c4e0d80e60aabbf61e79ab961 2023-03-21 11:58:23,568 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. as already flushing 2023-03-21 11:58:23,583 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/A/d54167423e5542feaf1f17767b3df8c0 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/d54167423e5542feaf1f17767b3df8c0 2023-03-21 11:58:23,586 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in cc711a8c4e0d80e60aabbf61e79ab961/A of cc711a8c4e0d80e60aabbf61e79ab961 into d54167423e5542feaf1f17767b3df8c0(size=26.0 K), total size for store is 51.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-21 11:58:23,586 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for cc711a8c4e0d80e60aabbf61e79ab961: 2023-03-21 11:58:23,586 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961., storeName=cc711a8c4e0d80e60aabbf61e79ab961/A, priority=13, startTime=1679399902754; duration=0sec 2023-03-21 11:58:23,586 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-21 11:58:23,596 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:23,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55146 deadline: 1679399963592, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:23,596 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:23,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55130 deadline: 1679399963593, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:23,694 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:23,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55174 deadline: 1679399963693, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:23,697 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:23,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55178 deadline: 1679399963695, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:23,697 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:23,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55146 deadline: 1679399963696, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:23,697 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:23,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55112 deadline: 1679399963696, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:23,698 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:23,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55130 deadline: 1679399963697, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:23,842 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230321beaacd762fe2469296c3156d7c3496c7 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230321beaacd762fe2469296c3156d7c3496c7 2023-03-21 11:58:23,848 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=314, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/A/f904ba169550474a851849f7e4428ecd 2023-03-21 11:58:23,854 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=314 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/B/1bf9ab7e8abd4de185e33639afdc9c49 2023-03-21 11:58:23,861 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=314 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/C/7f26821470f94801b2cfef7179895063 2023-03-21 11:58:23,864 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/A/f904ba169550474a851849f7e4428ecd as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/f904ba169550474a851849f7e4428ecd 2023-03-21 11:58:23,867 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/f904ba169550474a851849f7e4428ecd, entries=150, sequenceid=314, filesize=25.4 K 2023-03-21 11:58:23,868 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/B/1bf9ab7e8abd4de185e33639afdc9c49 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/1bf9ab7e8abd4de185e33639afdc9c49 2023-03-21 11:58:23,873 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/1bf9ab7e8abd4de185e33639afdc9c49, entries=150, sequenceid=314, filesize=11.9 K 2023-03-21 11:58:23,874 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/C/7f26821470f94801b2cfef7179895063 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/7f26821470f94801b2cfef7179895063 2023-03-21 11:58:23,879 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/7f26821470f94801b2cfef7179895063, entries=150, sequenceid=314, filesize=11.9 K 2023-03-21 11:58:23,880 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for cc711a8c4e0d80e60aabbf61e79ab961 in 447ms, sequenceid=314, compaction requested=true 2023-03-21 11:58:23,880 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for cc711a8c4e0d80e60aabbf61e79ab961: 2023-03-21 11:58:23,880 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:23,880 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:58:23,880 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:58:23,880 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:23,880 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:58:23,880 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:58:23,881 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:23,881 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:23,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:23,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:23,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:23,881 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:23,881 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:58:23,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:23,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:23,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:23,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:23,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:23,882 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:58:23,882 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d4166b8[Count = 0] remaining members to acquire global barrier 2023-03-21 11:58:23,882 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:58:23,882 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:23,883 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:23,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:23,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:23,883 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:58:23,883 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:58:23,883 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:58:23,883 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:23,883 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:58:23,884 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:23,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:23,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:23,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:23,884 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:58:23,884 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:23,884 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:58:23,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:23,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:23,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:23,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:23,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:23,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:23,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:23,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:58:23,887 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:58:23,887 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:58:23,887 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:58:23,887 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:58:23,887 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:58:23,887 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:58:23,888 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:23,888 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:23,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:23,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:23,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:23,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:23,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:23,888 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:23,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:23,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:23,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:23,889 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:23,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:23,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:23,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:23,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:23,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:23,890 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:23,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:23,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:23,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:23,892 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:23,892 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:23,892 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:23,894 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:23,894 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:23,894 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:23,894 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:23,894 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:58:23,894 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:58:23,894 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:23,894 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:23,894 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:23,894 DEBUG [Thread-3389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:58:23,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:23,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:23,894 DEBUG [Thread-3389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:58:23,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:23,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:23,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(9158): Flush requested on cc711a8c4e0d80e60aabbf61e79ab961 2023-03-21 11:58:23,900 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing cc711a8c4e0d80e60aabbf61e79ab961 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-21 11:58:23,901 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cc711a8c4e0d80e60aabbf61e79ab961, store=A 2023-03-21 11:58:23,901 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:58:23,901 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cc711a8c4e0d80e60aabbf61e79ab961, store=B 2023-03-21 11:58:23,901 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:58:23,901 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cc711a8c4e0d80e60aabbf61e79ab961, store=C 2023-03-21 11:58:23,901 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:58:23,934 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:23,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55130 deadline: 1679399963932, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:23,934 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:23,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55146 deadline: 1679399963933, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:23,995 DEBUG [Thread-3389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:58:23,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:58:23,996 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:58:23,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:58:23,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:58:23,997 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:58:23,997 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:23,997 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:58:23,997 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:58:23,997 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:23,998 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:23,998 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:23,998 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:23,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:23,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:23,998 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:23,998 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:58:23,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:23,999 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:23,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:58:23,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:23,999 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:58:23,999 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:58:23,999 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:58:23,999 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:23,999 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:58:23,999 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:58:23,999 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:58:23,999 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:23,999 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. started... 2023-03-21 11:58:23,999 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. as already flushing 2023-03-21 11:58:23,999 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:24,000 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:58:24,000 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:58:24,000 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,000 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:58:24,000 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:58:24,000 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,000 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:24,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:24,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:24,001 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:24,001 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:58:24,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:24,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:24,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:24,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:24,002 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:58:24,002 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3439ab9b[Count = 0] remaining members to acquire global barrier 2023-03-21 11:58:24,002 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:58:24,002 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:24,002 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:24,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:24,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:24,002 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:58:24,002 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,002 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:58:24,002 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:58:24,002 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:58:24,003 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,003 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:58:24,003 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:24,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:24,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:24,003 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:58:24,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:24,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:24,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:24,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:24,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:24,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:58:24,004 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:58:24,004 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:58:24,005 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:58:24,005 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:58:24,005 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:58:24,005 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:58:24,005 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:24,005 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:24,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:24,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:24,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:24,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:24,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:24,005 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:24,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:24,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:24,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:24,006 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:24,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:24,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:24,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:24,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,007 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:24,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:24,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,008 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,008 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:24,008 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:24,008 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,008 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:58:24,008 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:24,008 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:58:24,008 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:24,008 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:24,008 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:24,008 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:24,008 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:24,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:24,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:24,008 DEBUG [Thread-3389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:58:24,008 DEBUG [Thread-3389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:58:24,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:24,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:24,037 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:24,037 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:24,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55130 deadline: 1679399964035, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55146 deadline: 1679399964035, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,108 DEBUG [Thread-3389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:58:24,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:58:24,110 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:58:24,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:58:24,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:58:24,111 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:58:24,111 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:24,111 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:58:24,111 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:58:24,111 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:24,111 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:24,112 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,112 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:24,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:24,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:24,112 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,112 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:58:24,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:24,112 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:24,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:58:24,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:24,113 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:58:24,113 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:58:24,113 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:58:24,113 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:58:24,113 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:24,113 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:58:24,113 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:58:24,113 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:58:24,113 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:24,113 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. started... 2023-03-21 11:58:24,113 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. as already flushing 2023-03-21 11:58:24,113 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:24,113 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:58:24,113 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:58:24,113 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,113 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:58:24,113 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:58:24,114 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:24,114 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:24,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:24,114 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:24,114 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:58:24,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:24,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:24,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:24,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:24,116 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:58:24,116 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@bbcf1c6[Count = 0] remaining members to acquire global barrier 2023-03-21 11:58:24,116 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:58:24,116 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:24,117 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:24,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:24,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:24,117 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,117 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:58:24,117 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:58:24,117 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:58:24,117 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:58:24,117 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,118 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:58:24,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:24,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:24,118 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:24,118 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:58:24,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:24,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:24,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:24,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:24,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:24,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:58:24,121 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:58:24,121 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:58:24,121 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:58:24,121 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:58:24,121 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:58:24,121 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:58:24,122 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:24,122 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:24,122 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:24,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:24,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:24,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:24,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:24,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:24,122 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:24,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:24,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:24,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:24,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:24,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:24,123 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:24,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:24,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:24,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,124 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,124 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:24,124 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:24,124 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:24,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:24,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:24,124 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:58:24,124 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:58:24,124 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:24,124 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:24,124 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,125 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:24,125 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:24,125 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:24,125 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:58:24,125 DEBUG [Thread-3389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:58:24,125 DEBUG [Thread-3389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:58:24,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:24,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:24,225 DEBUG [Thread-3389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:58:24,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:58:24,226 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:58:24,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:58:24,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:58:24,228 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:58:24,228 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:24,228 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:58:24,228 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:58:24,228 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:24,228 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:24,229 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,229 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:24,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:24,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:24,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:24,230 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:24,230 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,230 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:58:24,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:58:24,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:24,230 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:58:24,230 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:58:24,230 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:58:24,230 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:58:24,230 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:24,230 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:58:24,231 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:58:24,231 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:24,231 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:58:24,231 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. started... 2023-03-21 11:58:24,231 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. as already flushing 2023-03-21 11:58:24,231 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:24,231 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:58:24,231 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:58:24,231 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,231 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:58:24,231 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:58:24,232 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:24,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:24,232 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:24,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:24,233 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:24,233 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:58:24,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:24,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:24,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:24,233 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:58:24,234 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5fa1adca[Count = 0] remaining members to acquire global barrier 2023-03-21 11:58:24,234 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:58:24,234 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:24,234 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:24,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:24,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:24,234 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:58:24,234 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:58:24,234 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:58:24,234 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,234 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:58:24,235 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:58:24,235 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:24,235 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,235 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:58:24,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:24,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:24,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:24,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:24,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:24,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:24,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:24,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:58:24,237 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:58:24,237 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:58:24,237 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:58:24,237 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:58:24,237 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:58:24,237 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:58:24,237 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:24,237 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:24,238 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:24,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:24,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:24,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:24,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:24,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:24,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:24,238 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:24,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:24,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:24,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:24,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:24,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:24,239 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:24,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:24,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,241 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:24,241 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:24,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55146 deadline: 1679399964238, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55130 deadline: 1679399964239, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,241 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:24,241 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:24,241 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:24,241 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:24,241 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:58:24,241 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:58:24,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:24,241 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:24,241 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:24,241 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,241 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:24,241 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:58:24,241 DEBUG [Thread-3389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:58:24,241 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:24,242 DEBUG [Thread-3389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:58:24,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:24,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:24,242 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:24,333 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303213162ff26e36a4a339768affa9b1ad23d to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303213162ff26e36a4a339768affa9b1ad23d 2023-03-21 11:58:24,337 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=335, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/A/3b7dd35ee0f64d32a6a127afcdc0189e 2023-03-21 11:58:24,342 DEBUG [Thread-3389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:58:24,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:58:24,343 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:58:24,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:58:24,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:58:24,344 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:58:24,344 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:24,344 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:58:24,344 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:58:24,345 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:24,345 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:24,345 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,345 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:24,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:24,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:24,345 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,345 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:58:24,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:24,346 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:24,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:58:24,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:24,346 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:58:24,346 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:58:24,346 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:58:24,346 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:58:24,346 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:24,346 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:58:24,347 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:58:24,347 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:58:24,347 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:24,347 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. started... 2023-03-21 11:58:24,347 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. as already flushing 2023-03-21 11:58:24,347 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:24,347 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:58:24,347 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:58:24,347 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,347 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:58:24,347 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:58:24,348 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,348 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:24,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:24,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:24,348 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:24,348 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:58:24,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:24,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:24,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:24,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:24,349 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:58:24,349 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@79f36420[Count = 0] remaining members to acquire global barrier 2023-03-21 11:58:24,349 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:58:24,349 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:24,350 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:24,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:24,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:24,350 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:58:24,350 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:58:24,350 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:58:24,350 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,350 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:58:24,351 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,351 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:58:24,351 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:24,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:24,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:24,351 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:58:24,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:24,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:24,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:24,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:24,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:24,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:58:24,352 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:58:24,352 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:58:24,352 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:58:24,352 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:58:24,352 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:58:24,352 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:58:24,353 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:24,353 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:24,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:24,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:24,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:24,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:24,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:24,353 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:24,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:24,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:24,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:24,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:24,354 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:24,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:24,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:24,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,355 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:24,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:24,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,356 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,356 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:24,356 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:58:24,356 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:58:24,356 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:24,356 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:24,356 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:24,356 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:24,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:24,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:24,356 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,356 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:24,356 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:24,356 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:24,356 DEBUG [Thread-3389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:58:24,357 DEBUG [Thread-3389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:58:24,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:24,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:24,457 DEBUG [Thread-3389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:58:24,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:58:24,458 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:58:24,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:58:24,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:58:24,459 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:58:24,459 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:24,459 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:58:24,459 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:58:24,460 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:24,460 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:24,460 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,460 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:24,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:24,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:24,460 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,460 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:58:24,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:24,461 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:24,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:58:24,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:24,461 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:58:24,461 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:58:24,461 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:58:24,461 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:58:24,461 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:24,461 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:58:24,461 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:58:24,461 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:58:24,461 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:24,461 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. started... 2023-03-21 11:58:24,461 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. as already flushing 2023-03-21 11:58:24,461 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:24,461 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:58:24,462 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:58:24,462 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,462 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:58:24,462 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:58:24,462 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:24,462 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:24,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:24,462 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:24,462 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:58:24,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:24,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:24,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:24,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:24,463 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:58:24,464 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f6df6[Count = 0] remaining members to acquire global barrier 2023-03-21 11:58:24,464 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:58:24,464 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:24,464 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:24,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:24,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:24,464 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:58:24,464 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:58:24,464 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,464 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:58:24,464 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:58:24,465 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:58:24,465 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:24,465 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,465 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:58:24,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:24,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:24,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:24,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:24,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:24,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:24,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:24,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:58:24,467 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:58:24,467 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:58:24,467 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:58:24,467 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:58:24,467 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:58:24,467 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:58:24,468 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:24,468 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:24,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:24,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:24,468 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:24,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:24,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:24,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:24,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:24,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:24,468 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:24,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:24,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:24,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:24,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:24,469 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:24,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:24,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,470 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:24,470 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:24,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:24,470 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:58:24,470 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:58:24,470 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:24,470 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:24,470 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:24,470 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:24,471 DEBUG [Thread-3389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:58:24,471 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:58:24,471 DEBUG [Thread-3389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:58:24,471 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:24,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:24,471 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:24,471 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:24,471 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:24,544 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:24,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55146 deadline: 1679399964544, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,545 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:24,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55130 deadline: 1679399964544, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,571 DEBUG [Thread-3389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:58:24,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:58:24,572 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:58:24,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:58:24,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:58:24,574 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:58:24,574 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:24,574 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:58:24,574 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:58:24,574 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:24,574 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:24,575 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,575 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:24,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:24,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:24,575 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,575 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:58:24,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:24,575 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:24,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:58:24,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:24,576 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:58:24,576 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:58:24,576 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:58:24,576 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:58:24,576 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:24,576 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:58:24,576 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:58:24,576 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:58:24,576 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:24,576 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. started... 2023-03-21 11:58:24,576 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. as already flushing 2023-03-21 11:58:24,576 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:24,576 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:58:24,576 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:58:24,576 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,576 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:58:24,576 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:58:24,577 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:24,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:24,577 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:24,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:24,577 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:24,577 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:58:24,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:24,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:24,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:24,578 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:58:24,578 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c319efc[Count = 0] remaining members to acquire global barrier 2023-03-21 11:58:24,578 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:58:24,578 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:24,579 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:24,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:24,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:24,579 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:58:24,579 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:58:24,579 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:58:24,579 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,579 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:58:24,580 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,580 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:58:24,580 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:24,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:24,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:24,580 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:58:24,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:24,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:24,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:24,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:24,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:24,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:58:24,582 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:58:24,582 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:58:24,582 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:58:24,582 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:58:24,582 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:58:24,582 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:58:24,583 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:24,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:24,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:24,583 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:24,583 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:24,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:24,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:24,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:24,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:24,584 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:24,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:24,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:24,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:24,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:24,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:24,585 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:24,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:24,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,587 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,587 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:24,587 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:24,587 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:24,587 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,587 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:24,587 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:24,587 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:24,587 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:24,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:24,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:24,587 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:58:24,587 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:58:24,587 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:24,587 DEBUG [Thread-3389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:58:24,587 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:58:24,587 DEBUG [Thread-3389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:58:24,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:24,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:24,688 DEBUG [Thread-3389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:58:24,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:58:24,693 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:58:24,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:58:24,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:58:24,698 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:58:24,698 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:24,698 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:58:24,698 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:58:24,699 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:24,699 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:24,701 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:24,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:24,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:24,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:24,701 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,702 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:24,702 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,702 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:58:24,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:58:24,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:24,703 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:58:24,710 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:58:24,710 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:58:24,710 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:58:24,710 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:24,710 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:58:24,714 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:58:24,714 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:24,714 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:58:24,714 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. started... 2023-03-21 11:58:24,715 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. as already flushing 2023-03-21 11:58:24,715 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:24,715 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:58:24,715 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:58:24,715 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,715 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:58:24,715 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:58:24,716 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,716 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:24,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:24,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:24,716 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:24,716 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:58:24,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:24,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:24,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:24,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:24,717 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:58:24,717 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e2ca14d[Count = 0] remaining members to acquire global barrier 2023-03-21 11:58:24,717 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:58:24,717 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:24,718 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:24,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:24,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:24,718 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,718 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:58:24,718 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:58:24,718 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:58:24,718 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:58:24,719 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:58:24,719 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:24,719 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:58:24,719 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:24,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:24,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:24,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:24,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:24,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:24,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:24,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:58:24,721 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:58:24,721 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:58:24,721 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:58:24,721 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:58:24,721 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:58:24,721 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:58:24,722 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:24,722 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:24,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:24,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:24,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:24,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:24,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:24,722 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:24,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:24,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:24,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:24,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:24,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:24,723 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:24,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:24,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:24,724 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:24,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,725 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,725 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:24,725 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:24,725 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:24,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:24,725 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:24,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:24,725 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:58:24,726 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:58:24,725 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,725 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:24,726 DEBUG [Thread-3389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:58:24,726 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:24,726 DEBUG [Thread-3389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:58:24,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:24,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:24,726 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:24,726 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:24,745 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=335 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/B/9d9b087d975044229d60395298340f73 2023-03-21 11:58:24,826 DEBUG [Thread-3389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:58:24,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:58:24,829 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:58:24,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:58:24,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:58:24,838 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:58:24,838 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:24,838 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:58:24,838 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:58:24,838 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:24,838 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:24,839 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:24,839 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:24,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:24,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:24,840 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,840 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:58:24,840 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:24,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:58:24,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:24,840 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:58:24,840 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:58:24,840 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:58:24,840 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:58:24,841 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:24,841 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:58:24,841 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:58:24,841 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:58:24,841 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:24,841 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. started... 2023-03-21 11:58:24,841 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. as already flushing 2023-03-21 11:58:24,841 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:24,841 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:58:24,841 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:58:24,841 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,841 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:58:24,841 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:58:24,842 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:24,842 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:24,843 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:58:24,846 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:24,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:24,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:24,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:24,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:24,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:24,848 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:58:24,848 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c9c00cc[Count = 0] remaining members to acquire global barrier 2023-03-21 11:58:24,848 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:58:24,848 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:24,857 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:24,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:24,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:24,857 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:58:24,857 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:58:24,857 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:58:24,857 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,857 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:58:24,861 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:58:24,861 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:24,861 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:58:24,861 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:24,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:24,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:24,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:24,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:24,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:24,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:24,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:58:24,864 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:58:24,864 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:58:24,864 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:58:24,864 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:58:24,864 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:58:24,864 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:58:24,873 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:24,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:24,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:24,874 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:24,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:24,874 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:24,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:24,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:24,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:24,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:24,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:24,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:24,874 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:24,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:24,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:24,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,876 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:24,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:24,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,878 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,878 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:24,878 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:24,878 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:24,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:24,878 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:58:24,878 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:58:24,878 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:24,878 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:24,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:24,878 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,878 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:24,878 DEBUG [Thread-3389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:58:24,878 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:24,878 DEBUG [Thread-3389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:58:24,878 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:24,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:24,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:24,978 DEBUG [Thread-3389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:58:24,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:58:24,981 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:58:24,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:58:24,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:58:24,986 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:58:24,986 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:24,986 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:58:24,986 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:58:24,987 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:24,987 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:24,987 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:24,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:24,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:24,987 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:24,988 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,988 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:58:24,989 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:24,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:58:24,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:24,989 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:58:24,990 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:58:24,990 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:58:24,990 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:58:24,990 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:24,990 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:58:24,996 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:58:24,996 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:24,996 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:58:24,996 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. started... 2023-03-21 11:58:24,996 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. as already flushing 2023-03-21 11:58:24,996 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:24,996 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:58:24,997 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:58:24,997 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:24,997 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:58:24,997 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:58:25,003 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:25,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:25,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:25,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:25,003 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:25,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:25,003 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:25,004 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:58:25,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:25,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:25,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:25,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:25,005 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:58:25,005 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@27e7c674[Count = 0] remaining members to acquire global barrier 2023-03-21 11:58:25,006 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:58:25,006 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:25,007 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:25,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:25,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:25,007 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:58:25,007 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:58:25,007 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:58:25,007 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:25,007 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:58:25,008 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:58:25,008 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:25,008 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:25,008 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:58:25,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:25,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:25,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:25,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:25,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:25,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:25,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:25,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:25,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:25,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:25,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:58:25,013 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:58:25,013 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:58:25,013 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:58:25,013 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:58:25,013 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:58:25,013 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:58:25,019 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:25,019 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:25,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:25,019 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:25,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:25,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:25,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:25,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:25,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:25,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:25,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:25,020 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:25,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:25,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:25,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:25,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:25,021 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:25,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:25,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:25,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:25,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:25,023 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:25,023 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:25,023 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:25,023 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:25,023 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:25,024 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:25,024 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:25,024 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:25,023 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:25,024 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:58:25,024 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:58:25,024 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:58:25,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:25,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:25,024 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:25,024 DEBUG [Thread-3389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:58:25,024 DEBUG [Thread-3389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:58:25,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:25,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:25,049 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:25,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55130 deadline: 1679399965048, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:25,050 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:25,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55146 deadline: 1679399965048, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:25,124 DEBUG [Thread-3389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:58:25,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:58:25,126 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:58:25,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:58:25,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:58:25,128 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:58:25,128 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:25,128 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:58:25,128 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:58:25,129 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:25,129 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:25,129 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:25,129 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:25,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:25,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:25,129 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:25,129 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:58:25,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:25,130 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:25,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:58:25,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:25,135 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:58:25,135 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:58:25,135 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:58:25,135 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:58:25,135 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:25,135 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:58:25,138 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:58:25,138 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:58:25,138 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:25,138 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. started... 2023-03-21 11:58:25,138 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. as already flushing 2023-03-21 11:58:25,138 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:25,138 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:58:25,138 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:58:25,138 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:25,139 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:58:25,139 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:58:25,145 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:25,145 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:25,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:25,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:25,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:25,146 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:25,146 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:58:25,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:25,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:25,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:25,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:25,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:25,147 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:58:25,147 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@47c16a0a[Count = 0] remaining members to acquire global barrier 2023-03-21 11:58:25,147 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:58:25,147 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:25,148 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:25,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:25,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:25,148 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:58:25,148 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:58:25,148 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:58:25,148 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:25,148 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:58:25,149 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:25,149 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:58:25,150 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:25,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:25,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:25,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:25,150 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:58:25,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:25,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:25,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:25,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:25,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:25,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:25,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:25,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:58:25,156 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:58:25,156 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:58:25,156 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:58:25,156 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:58:25,156 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:58:25,156 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:58:25,161 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:25,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:25,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:25,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:25,161 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:25,161 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:25,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:25,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:25,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:25,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:25,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:25,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:25,162 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:25,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:25,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:25,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:25,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:25,164 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:25,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:25,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:25,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:25,167 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:25,167 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:25,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:25,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:25,167 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:25,167 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:58:25,167 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:58:25,167 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:25,167 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:25,167 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:25,167 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:25,168 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:25,168 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:25,167 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:25,168 DEBUG [Thread-3389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:58:25,168 DEBUG [Thread-3389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:58:25,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:25,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:25,176 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=335 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/C/3caf1f449d1e4ae6a9c1a0b79827d5a9 2023-03-21 11:58:25,182 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/A/3b7dd35ee0f64d32a6a127afcdc0189e as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/3b7dd35ee0f64d32a6a127afcdc0189e 2023-03-21 11:58:25,189 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/3b7dd35ee0f64d32a6a127afcdc0189e, entries=200, sequenceid=335, filesize=32.3 K 2023-03-21 11:58:25,190 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/B/9d9b087d975044229d60395298340f73 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/9d9b087d975044229d60395298340f73 2023-03-21 11:58:25,209 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/9d9b087d975044229d60395298340f73, entries=150, sequenceid=335, filesize=11.9 K 2023-03-21 11:58:25,218 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/C/3caf1f449d1e4ae6a9c1a0b79827d5a9 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/3caf1f449d1e4ae6a9c1a0b79827d5a9 2023-03-21 11:58:25,223 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/3caf1f449d1e4ae6a9c1a0b79827d5a9, entries=150, sequenceid=335, filesize=11.9 K 2023-03-21 11:58:25,224 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for cc711a8c4e0d80e60aabbf61e79ab961 in 1324ms, sequenceid=335, compaction requested=true 2023-03-21 11:58:25,225 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for cc711a8c4e0d80e60aabbf61e79ab961: 2023-03-21 11:58:25,225 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-21 11:58:25,225 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-21 11:58:25,225 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-21 11:58:25,225 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-21 11:58:25,225 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-21 11:58:25,232 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49324 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-21 11:58:25,232 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1912): cc711a8c4e0d80e60aabbf61e79ab961/B is initiating minor compaction (all files) 2023-03-21 11:58:25,232 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of cc711a8c4e0d80e60aabbf61e79ab961/B in TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:25,232 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/c3ba93acec384b11a8b1c70491964730, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/fb4972bf9dc445fa8d5a01e581753210, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/1bf9ab7e8abd4de185e33639afdc9c49, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/9d9b087d975044229d60395298340f73] into tmpdir=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp, totalSize=48.2 K 2023-03-21 11:58:25,234 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 111804 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-21 11:58:25,234 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1912): cc711a8c4e0d80e60aabbf61e79ab961/A is initiating minor compaction (all files) 2023-03-21 11:58:25,234 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegion(2259): Starting compaction of cc711a8c4e0d80e60aabbf61e79ab961/A in TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:25,234 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/d54167423e5542feaf1f17767b3df8c0, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/2020c8cf2586406f8b2bf6bd2b1c6859, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/f904ba169550474a851849f7e4428ecd, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/3b7dd35ee0f64d32a6a127afcdc0189e] into tmpdir=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp, totalSize=109.2 K 2023-03-21 11:58:25,235 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting c3ba93acec384b11a8b1c70491964730, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=280, earliestPutTs=1679399901572 2023-03-21 11:58:25,235 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting d54167423e5542feaf1f17767b3df8c0, keycount=150, bloomtype=ROW, size=26.0 K, encoding=NONE, compression=NONE, seqNum=280, earliestPutTs=1679399901572 2023-03-21 11:58:25,235 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting fb4972bf9dc445fa8d5a01e581753210, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=300, earliestPutTs=1679399902323 2023-03-21 11:58:25,235 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 2020c8cf2586406f8b2bf6bd2b1c6859, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=300, earliestPutTs=1679399902323 2023-03-21 11:58:25,236 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting f904ba169550474a851849f7e4428ecd, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=314, earliestPutTs=1679399902955 2023-03-21 11:58:25,236 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 1bf9ab7e8abd4de185e33639afdc9c49, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=314, earliestPutTs=1679399902955 2023-03-21 11:58:25,236 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 3b7dd35ee0f64d32a6a127afcdc0189e, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=335, earliestPutTs=1679399903573 2023-03-21 11:58:25,237 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 9d9b087d975044229d60395298340f73, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=335, earliestPutTs=1679399903591 2023-03-21 11:58:25,254 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] throttle.PressureAwareThroughputController(145): cc711a8c4e0d80e60aabbf61e79ab961#B#compaction#495 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 57.69 MB/second 2023-03-21 11:58:25,268 DEBUG [Thread-3389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:58:25,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:58:25,272 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:58:25,284 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] throttle.PressureAwareThroughputController(145): cc711a8c4e0d80e60aabbf61e79ab961#A#compaction#496 average throughput is 0.76 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 57.69 MB/second 2023-03-21 11:58:25,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:58:25,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:58:25,286 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:58:25,286 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:25,286 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:58:25,286 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:58:25,286 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:25,286 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:25,287 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:25,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:25,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:25,287 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:25,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:25,288 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:25,288 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:58:25,288 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:25,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:58:25,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:25,289 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:58:25,294 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:58:25,294 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:58:25,294 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:58:25,294 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:25,294 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:58:25,309 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:58:25,309 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:58:25,311 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:25,311 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. started... 2023-03-21 11:58:25,311 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing cc711a8c4e0d80e60aabbf61e79ab961 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-21 11:58:25,311 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cc711a8c4e0d80e60aabbf61e79ab961, store=A 2023-03-21 11:58:25,312 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:58:25,312 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cc711a8c4e0d80e60aabbf61e79ab961, store=B 2023-03-21 11:58:25,312 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:58:25,312 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cc711a8c4e0d80e60aabbf61e79ab961, store=C 2023-03-21 11:58:25,312 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:58:25,375 WARN [DataStreamer for file /user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/A/2391e2dca6c141e58c2608143d78c230] hdfs.DataStreamer(982): Caught exception java.lang.InterruptedException at java.lang.Object.wait(Native Method) at java.lang.Thread.join(Thread.java:1257) at java.lang.Thread.join(Thread.java:1331) at org.apache.hadoop.hdfs.DataStreamer.closeResponder(DataStreamer.java:980) at org.apache.hadoop.hdfs.DataStreamer.endBlock(DataStreamer.java:630) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:807) 2023-03-21 11:58:25,382 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/A/2391e2dca6c141e58c2608143d78c230 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/2391e2dca6c141e58c2608143d78c230 2023-03-21 11:58:25,387 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in cc711a8c4e0d80e60aabbf61e79ab961/A of cc711a8c4e0d80e60aabbf61e79ab961 into 2391e2dca6c141e58c2608143d78c230(size=26.1 K), total size for store is 26.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-21 11:58:25,387 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for cc711a8c4e0d80e60aabbf61e79ab961: 2023-03-21 11:58:25,387 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961., storeName=cc711a8c4e0d80e60aabbf61e79ab961/A, priority=12, startTime=1679399905225; duration=0sec 2023-03-21 11:58:25,387 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-21 11:58:25,387 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-21 11:58:25,388 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49324 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-21 11:58:25,388 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1912): cc711a8c4e0d80e60aabbf61e79ab961/C is initiating minor compaction (all files) 2023-03-21 11:58:25,388 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegion(2259): Starting compaction of cc711a8c4e0d80e60aabbf61e79ab961/C in TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:25,388 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/16ad84427f934446b251aefeab82a0aa, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/712739b02b2b4d918fea140377c092fd, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/7f26821470f94801b2cfef7179895063, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/3caf1f449d1e4ae6a9c1a0b79827d5a9] into tmpdir=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp, totalSize=48.2 K 2023-03-21 11:58:25,389 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 16ad84427f934446b251aefeab82a0aa, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=280, earliestPutTs=1679399901572 2023-03-21 11:58:25,389 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 712739b02b2b4d918fea140377c092fd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=300, earliestPutTs=1679399902323 2023-03-21 11:58:25,389 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 7f26821470f94801b2cfef7179895063, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=314, earliestPutTs=1679399902955 2023-03-21 11:58:25,390 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 3caf1f449d1e4ae6a9c1a0b79827d5a9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=335, earliestPutTs=1679399903591 2023-03-21 11:58:25,415 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] throttle.PressureAwareThroughputController(145): cc711a8c4e0d80e60aabbf61e79ab961#C#compaction#498 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 57.69 MB/second 2023-03-21 11:58:25,436 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/C/274868ef0d9f4d17b9d8f4d2b445fb58 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/274868ef0d9f4d17b9d8f4d2b445fb58 2023-03-21 11:58:25,443 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in cc711a8c4e0d80e60aabbf61e79ab961/C of cc711a8c4e0d80e60aabbf61e79ab961 into 274868ef0d9f4d17b9d8f4d2b445fb58(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-21 11:58:25,444 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for cc711a8c4e0d80e60aabbf61e79ab961: 2023-03-21 11:58:25,444 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961., storeName=cc711a8c4e0d80e60aabbf61e79ab961/C, priority=12, startTime=1679399905225; duration=0sec 2023-03-21 11:58:25,444 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-21 11:58:25,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(9158): Flush requested on cc711a8c4e0d80e60aabbf61e79ab961 2023-03-21 11:58:25,699 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. as already flushing 2023-03-21 11:58:25,716 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/B/8a57bec314844d658caf4d588624019f as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/8a57bec314844d658caf4d588624019f 2023-03-21 11:58:25,733 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in cc711a8c4e0d80e60aabbf61e79ab961/B of cc711a8c4e0d80e60aabbf61e79ab961 into 8a57bec314844d658caf4d588624019f(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-21 11:58:25,733 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for cc711a8c4e0d80e60aabbf61e79ab961: 2023-03-21 11:58:25,733 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961., storeName=cc711a8c4e0d80e60aabbf61e79ab961/B, priority=12, startTime=1679399905225; duration=0sec 2023-03-21 11:58:25,733 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-21 11:58:25,751 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:25,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55174 deadline: 1679399965749, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:25,751 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:25,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55178 deadline: 1679399965749, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:25,753 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:25,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55112 deadline: 1679399965750, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:25,759 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230321971d281fd0d0472c86debce5f4a52197 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230321971d281fd0d0472c86debce5f4a52197 2023-03-21 11:58:25,853 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:25,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55178 deadline: 1679399965852, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:25,854 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:25,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55174 deadline: 1679399965852, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:25,856 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:25,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55112 deadline: 1679399965854, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:26,051 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:26,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55130 deadline: 1679399966050, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:26,056 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:26,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55178 deadline: 1679399966055, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:26,056 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:26,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55146 deadline: 1679399966055, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:26,056 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:26,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55174 deadline: 1679399966056, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:26,058 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:26,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55112 deadline: 1679399966057, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:26,183 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=347, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/A/bdc767d0e74d45908baedf566a9955e3 2023-03-21 11:58:26,360 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:26,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55112 deadline: 1679399966358, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:26,360 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:26,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55178 deadline: 1679399966359, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:26,361 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:26,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55174 deadline: 1679399966359, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:26,591 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=347 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/B/62e78cc8d4d94725867b8cd319f13edd 2023-03-21 11:58:26,862 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:26,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55178 deadline: 1679399966861, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:26,865 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:26,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55174 deadline: 1679399966863, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:26,866 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:26,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55112 deadline: 1679399966865, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:27,014 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=347 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/C/60d2a67e571d4389998a7112e3d30c1b 2023-03-21 11:58:27,018 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/A/bdc767d0e74d45908baedf566a9955e3 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/bdc767d0e74d45908baedf566a9955e3 2023-03-21 11:58:27,021 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/bdc767d0e74d45908baedf566a9955e3, entries=150, sequenceid=347, filesize=25.4 K 2023-03-21 11:58:27,022 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/B/62e78cc8d4d94725867b8cd319f13edd as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/62e78cc8d4d94725867b8cd319f13edd 2023-03-21 11:58:27,030 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/62e78cc8d4d94725867b8cd319f13edd, entries=150, sequenceid=347, filesize=11.9 K 2023-03-21 11:58:27,031 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/C/60d2a67e571d4389998a7112e3d30c1b as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/60d2a67e571d4389998a7112e3d30c1b 2023-03-21 11:58:27,034 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/60d2a67e571d4389998a7112e3d30c1b, entries=150, sequenceid=347, filesize=11.9 K 2023-03-21 11:58:27,035 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for cc711a8c4e0d80e60aabbf61e79ab961 in 1724ms, sequenceid=347, compaction requested=false 2023-03-21 11:58:27,035 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for cc711a8c4e0d80e60aabbf61e79ab961: 2023-03-21 11:58:27,035 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:27,035 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:58:27,035 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:58:27,035 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:27,036 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:58:27,036 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:58:27,037 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:27,037 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:27,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:27,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:27,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:27,037 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:27,037 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:58:27,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:27,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:27,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:27,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:27,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:27,040 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:58:27,040 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c3c8d76[Count = 0] remaining members to acquire global barrier 2023-03-21 11:58:27,040 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:58:27,040 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:27,051 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:27,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:27,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:27,051 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:58:27,052 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:58:27,052 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:58:27,058 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:27,058 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:58:27,060 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:58:27,060 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:27,060 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:27,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:27,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:27,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:27,060 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:58:27,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:27,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:27,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:27,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:27,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:27,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:27,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:27,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:58:27,069 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:58:27,069 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:58:27,069 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:58:27,069 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:58:27,069 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:58:27,069 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:58:27,087 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:27,087 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:27,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:27,087 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:27,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:27,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:27,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:27,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:27,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:27,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:27,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:27,088 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:27,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:27,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:27,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:27,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:27,089 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:27,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:27,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:27,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:27,090 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:27,091 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:27,091 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:27,091 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:58:27,091 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:58:27,091 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:58:27,091 DEBUG [Thread-3389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:58:27,091 DEBUG [Thread-3389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:58:27,091 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:27,091 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:27,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:27,091 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:27,091 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:27,091 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:27,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:27,091 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:27,091 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:27,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:27,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:27,191 DEBUG [Thread-3389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:58:27,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:58:27,193 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:58:27,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:58:27,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:58:27,195 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:58:27,195 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:27,195 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:58:27,195 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:58:27,196 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:27,196 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:27,196 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:27,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:27,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:27,197 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:27,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:27,197 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:27,197 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:27,197 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:58:27,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:58:27,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:27,197 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:58:27,202 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:58:27,202 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:58:27,202 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:58:27,202 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:27,202 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:58:27,202 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:58:27,202 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:58:27,202 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:27,203 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. started... 2023-03-21 11:58:27,203 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing cc711a8c4e0d80e60aabbf61e79ab961 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-21 11:58:27,203 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cc711a8c4e0d80e60aabbf61e79ab961, store=A 2023-03-21 11:58:27,203 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:58:27,203 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cc711a8c4e0d80e60aabbf61e79ab961, store=B 2023-03-21 11:58:27,203 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:58:27,203 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cc711a8c4e0d80e60aabbf61e79ab961, store=C 2023-03-21 11:58:27,203 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:58:27,638 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303211ef2fa4ce65e40fcb9efd8d8e53d38d5 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303211ef2fa4ce65e40fcb9efd8d8e53d38d5 2023-03-21 11:58:27,656 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=370, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/A/5309652080544c37b86e1022a37f9b1b 2023-03-21 11:58:27,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(9158): Flush requested on cc711a8c4e0d80e60aabbf61e79ab961 2023-03-21 11:58:27,868 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. as already flushing 2023-03-21 11:58:27,881 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:27,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55112 deadline: 1679399967879, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:27,882 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:27,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55174 deadline: 1679399967881, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:27,883 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:27,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55178 deadline: 1679399967882, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:27,984 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:27,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55112 deadline: 1679399967982, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:27,986 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:27,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55174 deadline: 1679399967983, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:27,986 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:27,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55178 deadline: 1679399967984, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:28,065 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:28,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55130 deadline: 1679399968065, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:28,067 DEBUG [Thread-3385] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4134 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961., hostname=jenkins-hbase20.apache.org,34369,1679399732030, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-21 11:58:28,072 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:28,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55146 deadline: 1679399968071, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:28,074 DEBUG [Thread-3379] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4140 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961., hostname=jenkins-hbase20.apache.org,34369,1679399732030, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-21 11:58:28,075 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=370 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/B/5f3e57c9a67449f2a4dfbf6b9756c824 2023-03-21 11:58:28,083 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=370 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/C/882bf88dd8fa43ab92adc07d88905133 2023-03-21 11:58:28,087 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/A/5309652080544c37b86e1022a37f9b1b as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/5309652080544c37b86e1022a37f9b1b 2023-03-21 11:58:28,092 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/5309652080544c37b86e1022a37f9b1b, entries=150, sequenceid=370, filesize=25.4 K 2023-03-21 11:58:28,092 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/B/5f3e57c9a67449f2a4dfbf6b9756c824 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/5f3e57c9a67449f2a4dfbf6b9756c824 2023-03-21 11:58:28,095 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/5f3e57c9a67449f2a4dfbf6b9756c824, entries=150, sequenceid=370, filesize=11.9 K 2023-03-21 11:58:28,096 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/C/882bf88dd8fa43ab92adc07d88905133 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/882bf88dd8fa43ab92adc07d88905133 2023-03-21 11:58:28,100 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/882bf88dd8fa43ab92adc07d88905133, entries=150, sequenceid=370, filesize=11.9 K 2023-03-21 11:58:28,101 INFO [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=67.09 KB/68700 for cc711a8c4e0d80e60aabbf61e79ab961 in 898ms, sequenceid=370, compaction requested=true 2023-03-21 11:58:28,101 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for cc711a8c4e0d80e60aabbf61e79ab961: 2023-03-21 11:58:28,101 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:28,101 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:58:28,101 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:58:28,101 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:28,101 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:58:28,101 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:58:28,102 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:28,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:28,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:28,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:28,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:28,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:28,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:28,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:28,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:28,104 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:58:28,104 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@209a645d[Count = 0] remaining members to acquire global barrier 2023-03-21 11:58:28,104 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:58:28,105 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:28,105 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:28,105 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:58:28,110 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:28,110 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(162): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:28,110 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:28,110 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:58:28,110 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:58:28,110 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:58:28,110 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:58:28,111 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:28,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:28,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:28,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:28,112 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:58:28,112 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:28,112 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:58:28,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:28,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:28,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:28,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:28,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:28,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:28,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:28,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:58:28,114 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:58:28,114 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:58:28,114 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:58:28,114 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:58:28,114 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:58:28,114 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:58:28,115 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:28,115 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:28,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:28,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:28,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:28,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:28,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:28,115 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:28,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:28,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:28,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:28,116 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:28,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:28,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:28,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:28,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:28,117 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:28,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:28,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:28,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:28,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:28,119 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:28,119 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:28,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:28,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:28,120 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:28,120 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:28,120 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:28,120 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:28,120 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:28,120 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:28,120 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:28,120 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:28,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:28,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:28,123 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:28,123 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:58:28,123 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:58:28,123 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:58:28,123 DEBUG [Thread-3389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:58:28,123 DEBUG [Thread-3389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:58:28,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(9158): Flush requested on cc711a8c4e0d80e60aabbf61e79ab961 2023-03-21 11:58:28,189 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing cc711a8c4e0d80e60aabbf61e79ab961 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-03-21 11:58:28,190 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cc711a8c4e0d80e60aabbf61e79ab961, store=A 2023-03-21 11:58:28,190 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:58:28,190 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cc711a8c4e0d80e60aabbf61e79ab961, store=B 2023-03-21 11:58:28,190 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:58:28,190 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cc711a8c4e0d80e60aabbf61e79ab961, store=C 2023-03-21 11:58:28,191 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:58:28,201 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303212ad93339c1ed4ebf81038d1958b77808 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303212ad93339c1ed4ebf81038d1958b77808 2023-03-21 11:58:28,212 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:28,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55178 deadline: 1679399968209, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:28,212 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:28,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55174 deadline: 1679399968209, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:28,212 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:28,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55112 deadline: 1679399968210, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:28,223 DEBUG [Thread-3389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:58:28,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:58:28,225 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-21 11:58:28,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-21 11:58:28,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-21 11:58:28,226 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-21 11:58:28,226 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:28,226 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-21 11:58:28,226 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-21 11:58:28,226 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:28,226 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:28,227 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:28,227 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:28,227 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:28,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:28,227 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:28,227 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-21 11:58:28,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:28,228 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:28,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-21 11:58:28,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:28,228 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-21 11:58:28,228 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-21 11:58:28,228 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-21 11:58:28,228 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-21 11:58:28,228 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-21 11:58:28,228 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-21 11:58:28,228 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-21 11:58:28,228 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-21 11:58:28,228 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:28,229 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. started... 2023-03-21 11:58:28,229 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. as already flushing 2023-03-21 11:58:28,229 DEBUG [rs(jenkins-hbase20.apache.org,34369,1679399732030)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:28,229 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-21 11:58:28,229 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-21 11:58:28,229 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:28,229 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-21 11:58:28,229 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-21 11:58:28,229 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:28,229 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:28,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:28,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:28,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:28,230 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:28,230 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-21 11:58:28,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:28,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:28,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:28,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:28,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:28,231 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34369,1679399732030' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-21 11:58:28,231 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@39d7c25c[Count = 0] remaining members to acquire global barrier 2023-03-21 11:58:28,231 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-21 11:58:28,231 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:28,231 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:28,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:28,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:28,231 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-21 11:58:28,231 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-21 11:58:28,231 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34369,1679399732030' in zk 2023-03-21 11:58:28,232 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:28,232 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-21 11:58:28,233 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:28,233 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-21 11:58:28,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:28,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:28,233 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:28,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:28,234 DEBUG [member: 'jenkins-hbase20.apache.org,34369,1679399732030' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-21 11:58:28,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:28,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:28,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:28,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:28,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:28,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:28,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:28,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34369,1679399732030': 2023-03-21 11:58:28,235 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34369,1679399732030' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-21 11:58:28,235 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-21 11:58:28,235 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-21 11:58:28,235 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-21 11:58:28,235 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-21 11:58:28,235 INFO [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-21 11:58:28,236 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:28,236 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:28,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:28,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-21 11:58:28,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-21 11:58:28,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:28,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:28,236 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:28,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-21 11:58:28,236 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:28,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:28,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:28,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:28,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:28,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-21 11:58:28,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:28,237 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:28,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:28,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-21 11:58:28,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-21 11:58:28,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:28,238 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:28,239 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:28,239 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-21 11:58:28,239 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-21 11:58:28,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-21 11:58:28,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-21 11:58:28,239 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-21 11:58:28,239 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-21 11:58:28,239 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-21 11:58:28,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-21 11:58:28,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-21 11:58:28,239 DEBUG [(jenkins-hbase20.apache.org,34627,1679399731089)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-21 11:58:28,239 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:28,239 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:28,239 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-21 11:58:28,239 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-21 11:58:28,239 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-21 11:58:28,239 DEBUG [Thread-3389] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-21 11:58:28,239 DEBUG [Thread-3389] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-21 11:58:28,301 DEBUG [Thread-3396] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x27489815 to 127.0.0.1:61761 2023-03-21 11:58:28,301 DEBUG [Thread-3396] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-21 11:58:28,301 DEBUG [Thread-3394] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x74adc5bd to 127.0.0.1:61761 2023-03-21 11:58:28,301 DEBUG [Thread-3394] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-21 11:58:28,302 DEBUG [Thread-3398] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x33765ee2 to 127.0.0.1:61761 2023-03-21 11:58:28,302 DEBUG [Thread-3398] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-21 11:58:28,304 DEBUG [Thread-3390] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x407f33a2 to 127.0.0.1:61761 2023-03-21 11:58:28,304 DEBUG [Thread-3390] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-21 11:58:28,304 DEBUG [Thread-3392] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7dfddf81 to 127.0.0.1:61761 2023-03-21 11:58:28,304 DEBUG [Thread-3392] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-21 11:58:28,313 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:28,313 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:28,313 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:28,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34369] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55178 deadline: 1679399968313, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:28,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55112 deadline: 1679399968313, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:28,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55174 deadline: 1679399968313, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:28,339 DEBUG [Thread-3389] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-21 11:58:28,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-21 11:58:28,515 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:28,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55174 deadline: 1679399968514, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:28,515 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:28,515 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:28,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55112 deadline: 1679399968515, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:28,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55178 deadline: 1679399968515, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:28,609 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=386, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/A/706cd2f346534185b00b0bce28969187 2023-03-21 11:58:28,816 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:28,816 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:28,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34369] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55112 deadline: 1679399968816, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:28,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55178 deadline: 1679399968816, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:28,818 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:28,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55174 deadline: 1679399968817, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:29,017 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=386 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/B/b0a83978998649b89f28304ca8a7f97b 2023-03-21 11:58:29,319 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:29,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55178 deadline: 1679399969318, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:29,319 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:29,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55112 deadline: 1679399969319, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:29,320 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-21 11:58:29,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55174 deadline: 1679399969320, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:29,431 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=386 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/C/800bde55a9194f18b26c189d489236a7 2023-03-21 11:58:29,434 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/A/706cd2f346534185b00b0bce28969187 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/706cd2f346534185b00b0bce28969187 2023-03-21 11:58:29,439 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/706cd2f346534185b00b0bce28969187, entries=250, sequenceid=386, filesize=39.2 K 2023-03-21 11:58:29,440 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/B/b0a83978998649b89f28304ca8a7f97b as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/b0a83978998649b89f28304ca8a7f97b 2023-03-21 11:58:29,445 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/b0a83978998649b89f28304ca8a7f97b, entries=150, sequenceid=386, filesize=11.9 K 2023-03-21 11:58:29,445 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/C/800bde55a9194f18b26c189d489236a7 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/800bde55a9194f18b26c189d489236a7 2023-03-21 11:58:29,448 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/800bde55a9194f18b26c189d489236a7, entries=150, sequenceid=386, filesize=11.9 K 2023-03-21 11:58:29,449 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for cc711a8c4e0d80e60aabbf61e79ab961 in 1260ms, sequenceid=386, compaction requested=true 2023-03-21 11:58:29,449 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for cc711a8c4e0d80e60aabbf61e79ab961: 2023-03-21 11:58:29,449 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-21 11:58:29,449 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-21 11:58:29,449 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-21 11:58:29,449 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-21 11:58:29,449 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-21 11:58:29,450 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 118984 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-21 11:58:29,450 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49460 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-21 11:58:29,450 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1912): cc711a8c4e0d80e60aabbf61e79ab961/A is initiating minor compaction (all files) 2023-03-21 11:58:29,450 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1912): cc711a8c4e0d80e60aabbf61e79ab961/B is initiating minor compaction (all files) 2023-03-21 11:58:29,450 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegion(2259): Starting compaction of cc711a8c4e0d80e60aabbf61e79ab961/A in TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:29,450 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of cc711a8c4e0d80e60aabbf61e79ab961/B in TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:29,450 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/2391e2dca6c141e58c2608143d78c230, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/bdc767d0e74d45908baedf566a9955e3, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/5309652080544c37b86e1022a37f9b1b, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/706cd2f346534185b00b0bce28969187] into tmpdir=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp, totalSize=116.2 K 2023-03-21 11:58:29,450 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/8a57bec314844d658caf4d588624019f, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/62e78cc8d4d94725867b8cd319f13edd, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/5f3e57c9a67449f2a4dfbf6b9756c824, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/b0a83978998649b89f28304ca8a7f97b] into tmpdir=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp, totalSize=48.3 K 2023-03-21 11:58:29,450 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 2391e2dca6c141e58c2608143d78c230, keycount=150, bloomtype=ROW, size=26.1 K, encoding=NONE, compression=NONE, seqNum=335, earliestPutTs=1679399903591 2023-03-21 11:58:29,450 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 8a57bec314844d658caf4d588624019f, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=335, earliestPutTs=1679399903591 2023-03-21 11:58:29,450 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting bdc767d0e74d45908baedf566a9955e3, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=347, earliestPutTs=1679399903903 2023-03-21 11:58:29,450 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 62e78cc8d4d94725867b8cd319f13edd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=347, earliestPutTs=1679399903903 2023-03-21 11:58:29,451 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 5309652080544c37b86e1022a37f9b1b, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=370, earliestPutTs=1679399905745 2023-03-21 11:58:29,451 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 5f3e57c9a67449f2a4dfbf6b9756c824, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=370, earliestPutTs=1679399905745 2023-03-21 11:58:29,451 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] compactions.Compactor(207): Compacting 706cd2f346534185b00b0bce28969187, keycount=250, bloomtype=ROW, size=39.2 K, encoding=NONE, compression=NONE, seqNum=386, earliestPutTs=1679399907877 2023-03-21 11:58:29,451 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting b0a83978998649b89f28304ca8a7f97b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=386, earliestPutTs=1679399907877 2023-03-21 11:58:29,457 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] throttle.PressureAwareThroughputController(145): cc711a8c4e0d80e60aabbf61e79ab961#B#compaction#507 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 57.69 MB/second 2023-03-21 11:58:29,461 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] throttle.PressureAwareThroughputController(145): cc711a8c4e0d80e60aabbf61e79ab961#A#compaction#508 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 57.69 MB/second 2023-03-21 11:58:29,872 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/B/7a72d0248f5c4f638ad3f836426080a0 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/7a72d0248f5c4f638ad3f836426080a0 2023-03-21 11:58:29,875 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in cc711a8c4e0d80e60aabbf61e79ab961/B of cc711a8c4e0d80e60aabbf61e79ab961 into 7a72d0248f5c4f638ad3f836426080a0(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-21 11:58:29,876 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for cc711a8c4e0d80e60aabbf61e79ab961: 2023-03-21 11:58:29,876 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961., storeName=cc711a8c4e0d80e60aabbf61e79ab961/B, priority=12, startTime=1679399909449; duration=0sec 2023-03-21 11:58:29,876 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-21 11:58:29,876 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-21 11:58:29,877 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49460 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-21 11:58:29,877 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1912): cc711a8c4e0d80e60aabbf61e79ab961/C is initiating minor compaction (all files) 2023-03-21 11:58:29,877 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of cc711a8c4e0d80e60aabbf61e79ab961/C in TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:29,877 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/274868ef0d9f4d17b9d8f4d2b445fb58, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/60d2a67e571d4389998a7112e3d30c1b, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/882bf88dd8fa43ab92adc07d88905133, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/800bde55a9194f18b26c189d489236a7] into tmpdir=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp, totalSize=48.3 K 2023-03-21 11:58:29,878 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 274868ef0d9f4d17b9d8f4d2b445fb58, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=335, earliestPutTs=1679399903591 2023-03-21 11:58:29,878 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 60d2a67e571d4389998a7112e3d30c1b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=347, earliestPutTs=1679399903903 2023-03-21 11:58:29,878 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 882bf88dd8fa43ab92adc07d88905133, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=370, earliestPutTs=1679399905745 2023-03-21 11:58:29,878 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] compactions.Compactor(207): Compacting 800bde55a9194f18b26c189d489236a7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=386, earliestPutTs=1679399907877 2023-03-21 11:58:29,885 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] throttle.PressureAwareThroughputController(145): cc711a8c4e0d80e60aabbf61e79ab961#C#compaction#509 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 57.69 MB/second 2023-03-21 11:58:30,297 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/C/7c352a34d01f4d07a33c0d8072556fe3 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/7c352a34d01f4d07a33c0d8072556fe3 2023-03-21 11:58:30,300 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in cc711a8c4e0d80e60aabbf61e79ab961/C of cc711a8c4e0d80e60aabbf61e79ab961 into 7c352a34d01f4d07a33c0d8072556fe3(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-21 11:58:30,300 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for cc711a8c4e0d80e60aabbf61e79ab961: 2023-03-21 11:58:30,300 INFO [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961., storeName=cc711a8c4e0d80e60aabbf61e79ab961/C, priority=12, startTime=1679399909449; duration=0sec 2023-03-21 11:58:30,301 DEBUG [RS:0;jenkins-hbase20:34369-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-21 11:58:30,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34369] regionserver.HRegion(9158): Flush requested on cc711a8c4e0d80e60aabbf61e79ab961 2023-03-21 11:58:30,322 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing cc711a8c4e0d80e60aabbf61e79ab961 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-03-21 11:58:30,322 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cc711a8c4e0d80e60aabbf61e79ab961, store=A 2023-03-21 11:58:30,322 DEBUG [Thread-3387] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x27acfde4 to 127.0.0.1:61761 2023-03-21 11:58:30,322 DEBUG [Thread-3387] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-21 11:58:30,322 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:58:30,322 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cc711a8c4e0d80e60aabbf61e79ab961, store=B 2023-03-21 11:58:30,322 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:58:30,322 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cc711a8c4e0d80e60aabbf61e79ab961, store=C 2023-03-21 11:58:30,322 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:58:30,323 DEBUG [Thread-3383] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x085873d0 to 127.0.0.1:61761 2023-03-21 11:58:30,323 DEBUG [Thread-3383] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-21 11:58:30,326 DEBUG [Thread-3381] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x774f1998 to 127.0.0.1:61761 2023-03-21 11:58:30,327 DEBUG [Thread-3381] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-21 11:58:30,680 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/A/7eda1a7d1551456fa6d8c48100e888d3 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/7eda1a7d1551456fa6d8c48100e888d3 2023-03-21 11:58:30,684 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in cc711a8c4e0d80e60aabbf61e79ab961/A of cc711a8c4e0d80e60aabbf61e79ab961 into 7eda1a7d1551456fa6d8c48100e888d3(size=26.3 K), total size for store is 26.3 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-03-21 11:58:30,684 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for cc711a8c4e0d80e60aabbf61e79ab961: 2023-03-21 11:58:30,684 INFO [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961., storeName=cc711a8c4e0d80e60aabbf61e79ab961/A, priority=12, startTime=1679399909449; duration=1sec 2023-03-21 11:58:30,684 DEBUG [RS:0;jenkins-hbase20:34369-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-21 11:58:30,733 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303216fecf424e5df410db7b0de7a27d3ccda to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303216fecf424e5df410db7b0de7a27d3ccda 2023-03-21 11:58:31,138 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=405, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/A/fc3a1046ae52495da3b5d0629ee75dd4 2023-03-21 11:58:31,554 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=405 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/B/5400a55fbb6b4604907fff784493bc9a 2023-03-21 11:58:31,967 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=405 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/C/6d5d5a86fdb647609b86516366500327 2023-03-21 11:58:31,971 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/A/fc3a1046ae52495da3b5d0629ee75dd4 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/fc3a1046ae52495da3b5d0629ee75dd4 2023-03-21 11:58:31,974 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/fc3a1046ae52495da3b5d0629ee75dd4, entries=150, sequenceid=405, filesize=25.4 K 2023-03-21 11:58:31,974 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/B/5400a55fbb6b4604907fff784493bc9a as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/5400a55fbb6b4604907fff784493bc9a 2023-03-21 11:58:31,977 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/5400a55fbb6b4604907fff784493bc9a, entries=150, sequenceid=405, filesize=11.9 K 2023-03-21 11:58:31,978 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/C/6d5d5a86fdb647609b86516366500327 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/6d5d5a86fdb647609b86516366500327 2023-03-21 11:58:31,981 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/6d5d5a86fdb647609b86516366500327, entries=150, sequenceid=405, filesize=11.9 K 2023-03-21 11:58:31,981 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=13.42 KB/13740 for cc711a8c4e0d80e60aabbf61e79ab961 in 1659ms, sequenceid=405, compaction requested=false 2023-03-21 11:58:31,982 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for cc711a8c4e0d80e60aabbf61e79ab961: 2023-03-21 11:58:32,084 DEBUG [Thread-3379] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x395f9d24 to 127.0.0.1:61761 2023-03-21 11:58:32,084 DEBUG [Thread-3379] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-21 11:58:32,089 DEBUG [Thread-3385] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x768816e9 to 127.0.0.1:61761 2023-03-21 11:58:32,089 DEBUG [Thread-3385] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-21 11:58:32,089 INFO [Listener at localhost.localdomain/37475] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-03-21 11:58:32,089 INFO [Listener at localhost.localdomain/37475] hbase.AcidGuaranteesTestTool(394): wrote 90 2023-03-21 11:58:32,089 INFO [Listener at localhost.localdomain/37475] hbase.AcidGuaranteesTestTool(394): wrote 40 2023-03-21 11:58:32,089 INFO [Listener at localhost.localdomain/37475] hbase.AcidGuaranteesTestTool(394): wrote 43 2023-03-21 11:58:32,090 INFO [Listener at localhost.localdomain/37475] hbase.AcidGuaranteesTestTool(394): wrote 110 2023-03-21 11:58:32,090 INFO [Listener at localhost.localdomain/37475] hbase.AcidGuaranteesTestTool(394): wrote 40 2023-03-21 11:58:32,090 INFO [Listener at localhost.localdomain/37475] hbase.AcidGuaranteesTestTool(396): Readers: 2023-03-21 11:58:32,090 INFO [Listener at localhost.localdomain/37475] hbase.AcidGuaranteesTestTool(398): read 5930 2023-03-21 11:58:32,090 INFO [Listener at localhost.localdomain/37475] hbase.AcidGuaranteesTestTool(398): read 5571 2023-03-21 11:58:32,090 INFO [Listener at localhost.localdomain/37475] hbase.AcidGuaranteesTestTool(398): read 5536 2023-03-21 11:58:32,090 INFO [Listener at localhost.localdomain/37475] hbase.AcidGuaranteesTestTool(398): read 5895 2023-03-21 11:58:32,090 INFO [Listener at localhost.localdomain/37475] hbase.AcidGuaranteesTestTool(398): read 5533 2023-03-21 11:58:32,090 INFO [Listener at localhost.localdomain/37475] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-03-21 11:58:32,090 INFO [Listener at localhost.localdomain/37475] client.ConnectionImplementation(1959): Closing master protocol: MasterService 2023-03-21 11:58:32,090 DEBUG [Listener at localhost.localdomain/37475] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x43e540fb to 127.0.0.1:61761 2023-03-21 11:58:32,090 DEBUG [Listener at localhost.localdomain/37475] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-21 11:58:32,091 INFO [Listener at localhost.localdomain/37475] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-03-21 11:58:32,092 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.HMaster$11(2418): Client=jenkins//148.251.75.209 disable TestAcidGuarantees 2023-03-21 11:58:32,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure2.ProcedureExecutor(1029): Stored pid=62, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-03-21 11:58:32,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-03-21 11:58:32,097 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679399912097"}]},"ts":"1679399912097"} 2023-03-21 11:58:32,098 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-03-21 11:58:32,099 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-03-21 11:58:32,100 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=63, ppid=62, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=cc711a8c4e0d80e60aabbf61e79ab961, UNASSIGN}] 2023-03-21 11:58:32,102 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=63, ppid=62, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=cc711a8c4e0d80e60aabbf61e79ab961, UNASSIGN 2023-03-21 11:58:32,102 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=63 updating hbase:meta row=cc711a8c4e0d80e60aabbf61e79ab961, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:32,103 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679399912102"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1679399912102"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679399912102"}]},"ts":"1679399912102"} 2023-03-21 11:58:32,104 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=64, ppid=63, state=RUNNABLE; CloseRegionProcedure cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030}] 2023-03-21 11:58:32,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-03-21 11:58:32,256 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close cc711a8c4e0d80e60aabbf61e79ab961 2023-03-21 11:58:32,256 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing cc711a8c4e0d80e60aabbf61e79ab961, disabling compactions & flushes 2023-03-21 11:58:32,256 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:32,256 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:32,257 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. after waiting 0 ms 2023-03-21 11:58:32,257 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:32,257 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing cc711a8c4e0d80e60aabbf61e79ab961 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-03-21 11:58:32,257 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cc711a8c4e0d80e60aabbf61e79ab961, store=A 2023-03-21 11:58:32,257 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:58:32,257 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cc711a8c4e0d80e60aabbf61e79ab961, store=B 2023-03-21 11:58:32,257 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:58:32,257 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cc711a8c4e0d80e60aabbf61e79ab961, store=C 2023-03-21 11:58:32,257 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-21 11:58:32,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-03-21 11:58:32,671 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230321be45f13ba9564ed380d35ddee88680ac to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230321be45f13ba9564ed380d35ddee88680ac 2023-03-21 11:58:32,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-03-21 11:58:33,077 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=413, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/A/9f2fa0dfc7644bbbb2285909dd2d046e 2023-03-21 11:58:33,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-03-21 11:58:33,491 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=413 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/B/d6ece633ed274e8186b0b0e7e934ff4e 2023-03-21 11:58:33,906 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=413 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/C/7382c10f499a4c6dbc5a1b4617d28a60 2023-03-21 11:58:33,910 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/A/9f2fa0dfc7644bbbb2285909dd2d046e as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/9f2fa0dfc7644bbbb2285909dd2d046e 2023-03-21 11:58:33,920 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/9f2fa0dfc7644bbbb2285909dd2d046e, entries=150, sequenceid=413, filesize=25.4 K 2023-03-21 11:58:33,920 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/B/d6ece633ed274e8186b0b0e7e934ff4e as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/d6ece633ed274e8186b0b0e7e934ff4e 2023-03-21 11:58:33,924 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/d6ece633ed274e8186b0b0e7e934ff4e, entries=150, sequenceid=413, filesize=11.9 K 2023-03-21 11:58:33,925 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/.tmp/C/7382c10f499a4c6dbc5a1b4617d28a60 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/7382c10f499a4c6dbc5a1b4617d28a60 2023-03-21 11:58:33,928 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/7382c10f499a4c6dbc5a1b4617d28a60, entries=150, sequenceid=413, filesize=11.9 K 2023-03-21 11:58:33,928 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for cc711a8c4e0d80e60aabbf61e79ab961 in 1671ms, sequenceid=413, compaction requested=true 2023-03-21 11:58:33,929 DEBUG [StoreCloser-TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/9bee3a786dea4a26bf8ed4cb00ead6a8, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/9842576d1c3f4d029f7e0d4bd1a2b1c1, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/4240e15c12b04d3e81f24438d9a29734, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/92d9338ebb4740c481ddd1dcf985d71d, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/fe27dff1472d4ce59d37bb5d641342f8, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/c486aa6e3be947f9aa1c212c84045f2a, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/41db545ed1c544c88c15c6ce47280df2, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/f2c16c492d9046a396017eae634bbec2, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/a98a48d8082a4ca1b951afa24ad40869, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/b3bcf43bd78440f882303558ec4be526, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/defb923ec42b470cb29d18cd54b28bd4, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/292986e6656c4c3693e3a2f2f600a7d2, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/5c23338f1a8240e08d210963d0f06716, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/1c77846499a54017812796e39596f501, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/f0dcbc97bde34471a9a13e7be84e8055, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/ccb54acca49a438c84dc55e0f6bc5e6d, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/28c08eb034584514ba21d92b7100b425, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/d54167423e5542feaf1f17767b3df8c0, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/2020c8cf2586406f8b2bf6bd2b1c6859, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/f904ba169550474a851849f7e4428ecd, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/3b7dd35ee0f64d32a6a127afcdc0189e, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/2391e2dca6c141e58c2608143d78c230, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/bdc767d0e74d45908baedf566a9955e3, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/5309652080544c37b86e1022a37f9b1b, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/706cd2f346534185b00b0bce28969187] to archive 2023-03-21 11:58:33,930 DEBUG [StoreCloser-TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-21 11:58:33,931 DEBUG [StoreCloser-TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/9bee3a786dea4a26bf8ed4cb00ead6a8 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/9bee3a786dea4a26bf8ed4cb00ead6a8 2023-03-21 11:58:33,932 DEBUG [StoreCloser-TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/9842576d1c3f4d029f7e0d4bd1a2b1c1 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/9842576d1c3f4d029f7e0d4bd1a2b1c1 2023-03-21 11:58:33,932 DEBUG [StoreCloser-TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/4240e15c12b04d3e81f24438d9a29734 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/4240e15c12b04d3e81f24438d9a29734 2023-03-21 11:58:33,933 DEBUG [StoreCloser-TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/92d9338ebb4740c481ddd1dcf985d71d to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/92d9338ebb4740c481ddd1dcf985d71d 2023-03-21 11:58:33,934 DEBUG [StoreCloser-TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/fe27dff1472d4ce59d37bb5d641342f8 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/fe27dff1472d4ce59d37bb5d641342f8 2023-03-21 11:58:33,935 DEBUG [StoreCloser-TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/c486aa6e3be947f9aa1c212c84045f2a to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/c486aa6e3be947f9aa1c212c84045f2a 2023-03-21 11:58:33,936 DEBUG [StoreCloser-TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/41db545ed1c544c88c15c6ce47280df2 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/41db545ed1c544c88c15c6ce47280df2 2023-03-21 11:58:33,937 DEBUG [StoreCloser-TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/f2c16c492d9046a396017eae634bbec2 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/f2c16c492d9046a396017eae634bbec2 2023-03-21 11:58:33,938 DEBUG [StoreCloser-TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/a98a48d8082a4ca1b951afa24ad40869 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/a98a48d8082a4ca1b951afa24ad40869 2023-03-21 11:58:33,938 DEBUG [StoreCloser-TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/b3bcf43bd78440f882303558ec4be526 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/b3bcf43bd78440f882303558ec4be526 2023-03-21 11:58:33,939 DEBUG [StoreCloser-TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/defb923ec42b470cb29d18cd54b28bd4 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/defb923ec42b470cb29d18cd54b28bd4 2023-03-21 11:58:33,940 DEBUG [StoreCloser-TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/292986e6656c4c3693e3a2f2f600a7d2 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/292986e6656c4c3693e3a2f2f600a7d2 2023-03-21 11:58:33,941 DEBUG [StoreCloser-TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/5c23338f1a8240e08d210963d0f06716 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/5c23338f1a8240e08d210963d0f06716 2023-03-21 11:58:33,942 DEBUG [StoreCloser-TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/1c77846499a54017812796e39596f501 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/1c77846499a54017812796e39596f501 2023-03-21 11:58:33,942 DEBUG [StoreCloser-TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/f0dcbc97bde34471a9a13e7be84e8055 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/f0dcbc97bde34471a9a13e7be84e8055 2023-03-21 11:58:33,943 DEBUG [StoreCloser-TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/ccb54acca49a438c84dc55e0f6bc5e6d to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/ccb54acca49a438c84dc55e0f6bc5e6d 2023-03-21 11:58:33,944 DEBUG [StoreCloser-TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/28c08eb034584514ba21d92b7100b425 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/28c08eb034584514ba21d92b7100b425 2023-03-21 11:58:33,945 DEBUG [StoreCloser-TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/d54167423e5542feaf1f17767b3df8c0 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/d54167423e5542feaf1f17767b3df8c0 2023-03-21 11:58:33,946 DEBUG [StoreCloser-TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/2020c8cf2586406f8b2bf6bd2b1c6859 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/2020c8cf2586406f8b2bf6bd2b1c6859 2023-03-21 11:58:33,946 DEBUG [StoreCloser-TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/f904ba169550474a851849f7e4428ecd to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/f904ba169550474a851849f7e4428ecd 2023-03-21 11:58:33,947 DEBUG [StoreCloser-TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/3b7dd35ee0f64d32a6a127afcdc0189e to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/3b7dd35ee0f64d32a6a127afcdc0189e 2023-03-21 11:58:33,948 DEBUG [StoreCloser-TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/2391e2dca6c141e58c2608143d78c230 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/2391e2dca6c141e58c2608143d78c230 2023-03-21 11:58:33,949 DEBUG [StoreCloser-TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/bdc767d0e74d45908baedf566a9955e3 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/bdc767d0e74d45908baedf566a9955e3 2023-03-21 11:58:33,949 DEBUG [StoreCloser-TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/5309652080544c37b86e1022a37f9b1b to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/5309652080544c37b86e1022a37f9b1b 2023-03-21 11:58:33,950 DEBUG [StoreCloser-TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/706cd2f346534185b00b0bce28969187 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/706cd2f346534185b00b0bce28969187 2023-03-21 11:58:33,953 DEBUG [StoreCloser-TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/ab7aacb9bff24b1aa02f840ba3d309ed, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/709cf3f129ee491ca2dce049878626f7, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/62536a8df9444844907c95116eb74784, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/3d19e94189124e4cbc4e47a2465b3629, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/62399949fee641a395f900f5c357b5d9, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/f586393140554ad19d819ac9da0c00fe, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/c7cb4612afbc4523b757d4c69a823c1b, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/ec0c96c278464a37b27af8afea396d1e, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/3e379f97af55485191c99a71435d9a11, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/e8b79d7e121f4c82b00768a168e3334f, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/8918745ddf7e412d986b8d3b68703819, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/c6c884a4724648c88009f1b520daa5a1, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/af95ed8c42ca478cb3e7dce00bc46237, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/078103d4734849d4a11a964db01a0135, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/bd6064f55aa54d9390d7b514465cf766, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/f69bacda2a984ccba2ac2e4dbd536e1d, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/c3ba93acec384b11a8b1c70491964730, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/4be497b2982d4a11b692214031e223f1, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/fb4972bf9dc445fa8d5a01e581753210, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/1bf9ab7e8abd4de185e33639afdc9c49, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/8a57bec314844d658caf4d588624019f, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/9d9b087d975044229d60395298340f73, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/62e78cc8d4d94725867b8cd319f13edd, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/5f3e57c9a67449f2a4dfbf6b9756c824, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/b0a83978998649b89f28304ca8a7f97b] to archive 2023-03-21 11:58:33,953 DEBUG [StoreCloser-TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-21 11:58:33,954 DEBUG [StoreCloser-TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/ab7aacb9bff24b1aa02f840ba3d309ed to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/ab7aacb9bff24b1aa02f840ba3d309ed 2023-03-21 11:58:33,955 DEBUG [StoreCloser-TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/709cf3f129ee491ca2dce049878626f7 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/709cf3f129ee491ca2dce049878626f7 2023-03-21 11:58:33,956 DEBUG [StoreCloser-TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/62536a8df9444844907c95116eb74784 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/62536a8df9444844907c95116eb74784 2023-03-21 11:58:33,956 DEBUG [StoreCloser-TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/3d19e94189124e4cbc4e47a2465b3629 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/3d19e94189124e4cbc4e47a2465b3629 2023-03-21 11:58:33,957 DEBUG [StoreCloser-TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/62399949fee641a395f900f5c357b5d9 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/62399949fee641a395f900f5c357b5d9 2023-03-21 11:58:33,958 DEBUG [StoreCloser-TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/f586393140554ad19d819ac9da0c00fe to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/f586393140554ad19d819ac9da0c00fe 2023-03-21 11:58:33,959 DEBUG [StoreCloser-TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/c7cb4612afbc4523b757d4c69a823c1b to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/c7cb4612afbc4523b757d4c69a823c1b 2023-03-21 11:58:33,959 DEBUG [StoreCloser-TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/ec0c96c278464a37b27af8afea396d1e to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/ec0c96c278464a37b27af8afea396d1e 2023-03-21 11:58:33,960 DEBUG [StoreCloser-TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/3e379f97af55485191c99a71435d9a11 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/3e379f97af55485191c99a71435d9a11 2023-03-21 11:58:33,961 DEBUG [StoreCloser-TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/e8b79d7e121f4c82b00768a168e3334f to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/e8b79d7e121f4c82b00768a168e3334f 2023-03-21 11:58:33,962 DEBUG [StoreCloser-TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/8918745ddf7e412d986b8d3b68703819 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/8918745ddf7e412d986b8d3b68703819 2023-03-21 11:58:33,963 DEBUG [StoreCloser-TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/c6c884a4724648c88009f1b520daa5a1 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/c6c884a4724648c88009f1b520daa5a1 2023-03-21 11:58:33,963 DEBUG [StoreCloser-TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/af95ed8c42ca478cb3e7dce00bc46237 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/af95ed8c42ca478cb3e7dce00bc46237 2023-03-21 11:58:33,964 DEBUG [StoreCloser-TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/078103d4734849d4a11a964db01a0135 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/078103d4734849d4a11a964db01a0135 2023-03-21 11:58:33,965 DEBUG [StoreCloser-TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/bd6064f55aa54d9390d7b514465cf766 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/bd6064f55aa54d9390d7b514465cf766 2023-03-21 11:58:33,966 DEBUG [StoreCloser-TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/f69bacda2a984ccba2ac2e4dbd536e1d to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/f69bacda2a984ccba2ac2e4dbd536e1d 2023-03-21 11:58:33,966 DEBUG [StoreCloser-TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/c3ba93acec384b11a8b1c70491964730 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/c3ba93acec384b11a8b1c70491964730 2023-03-21 11:58:33,967 DEBUG [StoreCloser-TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/4be497b2982d4a11b692214031e223f1 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/4be497b2982d4a11b692214031e223f1 2023-03-21 11:58:33,968 DEBUG [StoreCloser-TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/fb4972bf9dc445fa8d5a01e581753210 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/fb4972bf9dc445fa8d5a01e581753210 2023-03-21 11:58:33,969 DEBUG [StoreCloser-TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/1bf9ab7e8abd4de185e33639afdc9c49 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/1bf9ab7e8abd4de185e33639afdc9c49 2023-03-21 11:58:33,969 DEBUG [StoreCloser-TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/8a57bec314844d658caf4d588624019f to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/8a57bec314844d658caf4d588624019f 2023-03-21 11:58:33,970 DEBUG [StoreCloser-TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/9d9b087d975044229d60395298340f73 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/9d9b087d975044229d60395298340f73 2023-03-21 11:58:33,971 DEBUG [StoreCloser-TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/62e78cc8d4d94725867b8cd319f13edd to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/62e78cc8d4d94725867b8cd319f13edd 2023-03-21 11:58:33,972 DEBUG [StoreCloser-TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/5f3e57c9a67449f2a4dfbf6b9756c824 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/5f3e57c9a67449f2a4dfbf6b9756c824 2023-03-21 11:58:33,973 DEBUG [StoreCloser-TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/b0a83978998649b89f28304ca8a7f97b to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/b0a83978998649b89f28304ca8a7f97b 2023-03-21 11:58:33,975 DEBUG [StoreCloser-TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/ae2c3046634b4a2dbf982036bddb03d9, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/fa5e691f8c3b4f18b9a12e51e5392da5, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/2491db6eee204fdd84376f13fb75c149, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/937f89cb70ef470ab1405db22ca41972, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/4bcb4e67df8c4d97b62f9c129502c526, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/6045696af9744b4f91b718d1213352fc, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/9e5ec838908846808d55649fb1447353, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/6823865344744aed969482e603e43fa9, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/f9fd1189769a4e55a7b416911e3bb8a3, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/260e444f57bc496eaa872ac7cfaaf58d, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/57d09b7c6c164e71b25efd5aa897f8f1, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/e46a05e063434332b9847c59242eba44, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/47ffd81cd1cf481d865ab08a2d1330b9, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/7739b5dc3de04798a076d8be162b3aea, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/b948beeaa4184106bfe62a6aaee49c51, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/9f44ee270f3a4a4c95fe8792635a3688, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/16ad84427f934446b251aefeab82a0aa, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/895fb44174e64a1ba10b6979dd319430, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/712739b02b2b4d918fea140377c092fd, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/7f26821470f94801b2cfef7179895063, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/274868ef0d9f4d17b9d8f4d2b445fb58, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/3caf1f449d1e4ae6a9c1a0b79827d5a9, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/60d2a67e571d4389998a7112e3d30c1b, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/882bf88dd8fa43ab92adc07d88905133, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/800bde55a9194f18b26c189d489236a7] to archive 2023-03-21 11:58:33,976 DEBUG [StoreCloser-TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-21 11:58:33,977 DEBUG [StoreCloser-TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/ae2c3046634b4a2dbf982036bddb03d9 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/ae2c3046634b4a2dbf982036bddb03d9 2023-03-21 11:58:33,977 DEBUG [StoreCloser-TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/fa5e691f8c3b4f18b9a12e51e5392da5 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/fa5e691f8c3b4f18b9a12e51e5392da5 2023-03-21 11:58:33,978 DEBUG [StoreCloser-TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/2491db6eee204fdd84376f13fb75c149 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/2491db6eee204fdd84376f13fb75c149 2023-03-21 11:58:33,979 DEBUG [StoreCloser-TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/937f89cb70ef470ab1405db22ca41972 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/937f89cb70ef470ab1405db22ca41972 2023-03-21 11:58:33,980 DEBUG [StoreCloser-TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/4bcb4e67df8c4d97b62f9c129502c526 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/4bcb4e67df8c4d97b62f9c129502c526 2023-03-21 11:58:33,980 DEBUG [StoreCloser-TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/6045696af9744b4f91b718d1213352fc to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/6045696af9744b4f91b718d1213352fc 2023-03-21 11:58:33,981 DEBUG [StoreCloser-TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/9e5ec838908846808d55649fb1447353 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/9e5ec838908846808d55649fb1447353 2023-03-21 11:58:33,982 DEBUG [StoreCloser-TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/6823865344744aed969482e603e43fa9 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/6823865344744aed969482e603e43fa9 2023-03-21 11:58:33,983 DEBUG [StoreCloser-TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/f9fd1189769a4e55a7b416911e3bb8a3 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/f9fd1189769a4e55a7b416911e3bb8a3 2023-03-21 11:58:33,983 DEBUG [StoreCloser-TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/260e444f57bc496eaa872ac7cfaaf58d to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/260e444f57bc496eaa872ac7cfaaf58d 2023-03-21 11:58:33,984 DEBUG [StoreCloser-TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/57d09b7c6c164e71b25efd5aa897f8f1 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/57d09b7c6c164e71b25efd5aa897f8f1 2023-03-21 11:58:33,985 DEBUG [StoreCloser-TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/e46a05e063434332b9847c59242eba44 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/e46a05e063434332b9847c59242eba44 2023-03-21 11:58:33,986 DEBUG [StoreCloser-TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/47ffd81cd1cf481d865ab08a2d1330b9 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/47ffd81cd1cf481d865ab08a2d1330b9 2023-03-21 11:58:33,987 DEBUG [StoreCloser-TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/7739b5dc3de04798a076d8be162b3aea to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/7739b5dc3de04798a076d8be162b3aea 2023-03-21 11:58:33,987 DEBUG [StoreCloser-TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/b948beeaa4184106bfe62a6aaee49c51 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/b948beeaa4184106bfe62a6aaee49c51 2023-03-21 11:58:33,988 DEBUG [StoreCloser-TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/9f44ee270f3a4a4c95fe8792635a3688 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/9f44ee270f3a4a4c95fe8792635a3688 2023-03-21 11:58:33,989 DEBUG [StoreCloser-TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/16ad84427f934446b251aefeab82a0aa to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/16ad84427f934446b251aefeab82a0aa 2023-03-21 11:58:33,990 DEBUG [StoreCloser-TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/895fb44174e64a1ba10b6979dd319430 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/895fb44174e64a1ba10b6979dd319430 2023-03-21 11:58:33,991 DEBUG [StoreCloser-TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/712739b02b2b4d918fea140377c092fd to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/712739b02b2b4d918fea140377c092fd 2023-03-21 11:58:33,992 DEBUG [StoreCloser-TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/7f26821470f94801b2cfef7179895063 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/7f26821470f94801b2cfef7179895063 2023-03-21 11:58:33,993 DEBUG [StoreCloser-TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/274868ef0d9f4d17b9d8f4d2b445fb58 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/274868ef0d9f4d17b9d8f4d2b445fb58 2023-03-21 11:58:33,994 DEBUG [StoreCloser-TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/3caf1f449d1e4ae6a9c1a0b79827d5a9 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/3caf1f449d1e4ae6a9c1a0b79827d5a9 2023-03-21 11:58:33,995 DEBUG [StoreCloser-TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/60d2a67e571d4389998a7112e3d30c1b to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/60d2a67e571d4389998a7112e3d30c1b 2023-03-21 11:58:33,996 DEBUG [StoreCloser-TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/882bf88dd8fa43ab92adc07d88905133 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/882bf88dd8fa43ab92adc07d88905133 2023-03-21 11:58:33,997 DEBUG [StoreCloser-TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/800bde55a9194f18b26c189d489236a7 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/800bde55a9194f18b26c189d489236a7 2023-03-21 11:58:34,001 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/recovered.edits/416.seqid, newMaxSeqId=416, maxSeqId=4 2023-03-21 11:58:34,003 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961. 2023-03-21 11:58:34,003 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for cc711a8c4e0d80e60aabbf61e79ab961: 2023-03-21 11:58:34,004 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed cc711a8c4e0d80e60aabbf61e79ab961 2023-03-21 11:58:34,005 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=63 updating hbase:meta row=cc711a8c4e0d80e60aabbf61e79ab961, regionState=CLOSED 2023-03-21 11:58:34,005 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679399914005"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679399914005"}]},"ts":"1679399914005"} 2023-03-21 11:58:34,008 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=64, resume processing ppid=63 2023-03-21 11:58:34,008 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=64, ppid=63, state=SUCCESS; CloseRegionProcedure cc711a8c4e0d80e60aabbf61e79ab961, server=jenkins-hbase20.apache.org,34369,1679399732030 in 1.9020 sec 2023-03-21 11:58:34,010 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=63, resume processing ppid=62 2023-03-21 11:58:34,010 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=63, ppid=62, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=cc711a8c4e0d80e60aabbf61e79ab961, UNASSIGN in 1.9080 sec 2023-03-21 11:58:34,011 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679399914011"}]},"ts":"1679399914011"} 2023-03-21 11:58:34,012 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-03-21 11:58:34,013 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-03-21 11:58:34,015 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=62, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.9220 sec 2023-03-21 11:58:34,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-03-21 11:58:34,201 INFO [Listener at localhost.localdomain/37475] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 62 completed 2023-03-21 11:58:34,202 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.HMaster$5(2228): Client=jenkins//148.251.75.209 delete TestAcidGuarantees 2023-03-21 11:58:34,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] procedure2.ProcedureExecutor(1029): Stored pid=65, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-03-21 11:58:34,204 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=65, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-21 11:58:34,205 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=65, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-21 11:58:34,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-03-21 11:58:34,208 DEBUG [HFileArchiver-13] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/.tmp/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961 2023-03-21 11:58:34,210 DEBUG [HFileArchiver-13] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/.tmp/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A, FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/.tmp/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B, FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/.tmp/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C, FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/.tmp/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/recovered.edits] 2023-03-21 11:58:34,212 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/.tmp/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/7eda1a7d1551456fa6d8c48100e888d3 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/7eda1a7d1551456fa6d8c48100e888d3 2023-03-21 11:58:34,213 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/.tmp/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/9f2fa0dfc7644bbbb2285909dd2d046e to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/9f2fa0dfc7644bbbb2285909dd2d046e 2023-03-21 11:58:34,214 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/.tmp/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/fc3a1046ae52495da3b5d0629ee75dd4 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/A/fc3a1046ae52495da3b5d0629ee75dd4 2023-03-21 11:58:34,216 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/.tmp/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/5400a55fbb6b4604907fff784493bc9a to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/5400a55fbb6b4604907fff784493bc9a 2023-03-21 11:58:34,216 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/.tmp/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/7a72d0248f5c4f638ad3f836426080a0 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/7a72d0248f5c4f638ad3f836426080a0 2023-03-21 11:58:34,217 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/.tmp/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/d6ece633ed274e8186b0b0e7e934ff4e to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/B/d6ece633ed274e8186b0b0e7e934ff4e 2023-03-21 11:58:34,219 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/.tmp/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/6d5d5a86fdb647609b86516366500327 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/6d5d5a86fdb647609b86516366500327 2023-03-21 11:58:34,220 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/.tmp/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/7382c10f499a4c6dbc5a1b4617d28a60 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/7382c10f499a4c6dbc5a1b4617d28a60 2023-03-21 11:58:34,221 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/.tmp/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/7c352a34d01f4d07a33c0d8072556fe3 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/C/7c352a34d01f4d07a33c0d8072556fe3 2023-03-21 11:58:34,223 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/.tmp/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/recovered.edits/416.seqid to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961/recovered.edits/416.seqid 2023-03-21 11:58:34,223 DEBUG [HFileArchiver-13] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/.tmp/data/default/TestAcidGuarantees/cc711a8c4e0d80e60aabbf61e79ab961 2023-03-21 11:58:34,223 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-21 11:58:34,223 DEBUG [PEWorker-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-03-21 11:58:34,224 DEBUG [PEWorker-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-03-21 11:58:34,226 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303211b4330e84fa84291bb453d5b123a0403 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303211b4330e84fa84291bb453d5b123a0403 2023-03-21 11:58:34,227 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303211ef2fa4ce65e40fcb9efd8d8e53d38d5 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303211ef2fa4ce65e40fcb9efd8d8e53d38d5 2023-03-21 11:58:34,228 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303212ad93339c1ed4ebf81038d1958b77808 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303212ad93339c1ed4ebf81038d1958b77808 2023-03-21 11:58:34,228 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303212cf0b47938da4c198c78bd2cd118f807 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303212cf0b47938da4c198c78bd2cd118f807 2023-03-21 11:58:34,229 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303212ee8bfb8a437455d87312f2745e9365f to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303212ee8bfb8a437455d87312f2745e9365f 2023-03-21 11:58:34,230 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303213162ff26e36a4a339768affa9b1ad23d to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303213162ff26e36a4a339768affa9b1ad23d 2023-03-21 11:58:34,231 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230321329205302a41469496283ad325c1d341 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230321329205302a41469496283ad325c1d341 2023-03-21 11:58:34,231 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303213a2bed14280543148d0ef360c4a5a260 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303213a2bed14280543148d0ef360c4a5a260 2023-03-21 11:58:34,232 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303214d0f304aff7d4dd2858bec6fcaba27d5 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303214d0f304aff7d4dd2858bec6fcaba27d5 2023-03-21 11:58:34,232 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303214f25b8a27ef04386bbdab5edb13a9e2a to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303214f25b8a27ef04386bbdab5edb13a9e2a 2023-03-21 11:58:34,233 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303216fecf424e5df410db7b0de7a27d3ccda to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303216fecf424e5df410db7b0de7a27d3ccda 2023-03-21 11:58:34,234 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230321823b6446b9ed4d6b9d45b1e535bdf3b5 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230321823b6446b9ed4d6b9d45b1e535bdf3b5 2023-03-21 11:58:34,234 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303219561cf4b1cc24e8493c2c1ddebe6ef34 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303219561cf4b1cc24e8493c2c1ddebe6ef34 2023-03-21 11:58:34,235 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230321971d281fd0d0472c86debce5f4a52197 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230321971d281fd0d0472c86debce5f4a52197 2023-03-21 11:58:34,235 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230321980f5c523f9c439a9c1ecf97405b115e to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230321980f5c523f9c439a9c1ecf97405b115e 2023-03-21 11:58:34,236 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230321b6fbf3a7aaf645f9826c71a676328540 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230321b6fbf3a7aaf645f9826c71a676328540 2023-03-21 11:58:34,237 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230321bb5cf131faac429e91293dff402d4147 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230321bb5cf131faac429e91293dff402d4147 2023-03-21 11:58:34,237 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230321be45f13ba9564ed380d35ddee88680ac to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230321be45f13ba9564ed380d35ddee88680ac 2023-03-21 11:58:34,238 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230321beaacd762fe2469296c3156d7c3496c7 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230321beaacd762fe2469296c3156d7c3496c7 2023-03-21 11:58:34,239 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230321c8fe639d648442afbca3bb0d312444dc to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230321c8fe639d648442afbca3bb0d312444dc 2023-03-21 11:58:34,239 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230321cd9d22d502084078ab05a7a1a761fe80 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230321cd9d22d502084078ab05a7a1a761fe80 2023-03-21 11:58:34,240 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230321fd4ed93a970242feac034fbeaaae2452 to hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230321fd4ed93a970242feac034fbeaaae2452 2023-03-21 11:58:34,240 DEBUG [PEWorker-5] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-03-21 11:58:34,241 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=65, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-21 11:58:34,243 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-03-21 11:58:34,244 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-03-21 11:58:34,245 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=65, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-21 11:58:34,245 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-03-21 11:58:34,245 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1679399914245"}]},"ts":"9223372036854775807"} 2023-03-21 11:58:34,247 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-03-21 11:58:34,247 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => cc711a8c4e0d80e60aabbf61e79ab961, NAME => 'TestAcidGuarantees,,1679399885153.cc711a8c4e0d80e60aabbf61e79ab961.', STARTKEY => '', ENDKEY => ''}] 2023-03-21 11:58:34,247 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-03-21 11:58:34,247 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1679399914247"}]},"ts":"9223372036854775807"} 2023-03-21 11:58:34,248 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-03-21 11:58:34,249 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=65, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-21 11:58:34,250 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=65, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 47 msec 2023-03-21 11:58:34,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34627] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-03-21 11:58:34,306 INFO [Listener at localhost.localdomain/37475] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 65 completed 2023-03-21 11:58:34,316 INFO [Listener at localhost.localdomain/37475] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobGetAtomicity Thread=228 (was 228), OpenFileDescriptor=554 (was 550) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=338 (was 383), ProcessCount=166 (was 166), AvailableMemoryMB=9210 (was 9176) - AvailableMemoryMB LEAK? - 2023-03-21 11:58:34,317 INFO [Listener at localhost.localdomain/37475] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-03-21 11:58:34,317 INFO [Listener at localhost.localdomain/37475] client.ConnectionImplementation(1959): Closing master protocol: MasterService 2023-03-21 11:58:34,317 DEBUG [Listener at localhost.localdomain/37475] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3c41175f to 127.0.0.1:61761 2023-03-21 11:58:34,317 DEBUG [Listener at localhost.localdomain/37475] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-21 11:58:34,318 DEBUG [Listener at localhost.localdomain/37475] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-03-21 11:58:34,318 DEBUG [Listener at localhost.localdomain/37475] util.JVMClusterUtil(257): Found active master hash=954541380, stopped=false 2023-03-21 11:58:34,318 INFO [Listener at localhost.localdomain/37475] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase20.apache.org,34627,1679399731089 2023-03-21 11:58:34,319 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-03-21 11:58:34,319 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-03-21 11:58:34,319 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-21 11:58:34,319 INFO [Listener at localhost.localdomain/37475] procedure2.ProcedureExecutor(629): Stopping 2023-03-21 11:58:34,319 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-03-21 11:58:34,319 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-03-21 11:58:34,319 DEBUG [Listener at localhost.localdomain/37475] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x51e2041c to 127.0.0.1:61761 2023-03-21 11:58:34,319 DEBUG [Listener at localhost.localdomain/37475] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-21 11:58:34,319 INFO [Listener at localhost.localdomain/37475] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase20.apache.org,34369,1679399732030' ***** 2023-03-21 11:58:34,319 INFO [Listener at localhost.localdomain/37475] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-03-21 11:58:34,320 INFO [RS:0;jenkins-hbase20:34369] regionserver.HeapMemoryManager(220): Stopping 2023-03-21 11:58:34,320 INFO [RS:0;jenkins-hbase20:34369] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-03-21 11:58:34,320 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-03-21 11:58:34,320 INFO [RS:0;jenkins-hbase20:34369] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-03-21 11:58:34,320 INFO [RS:0;jenkins-hbase20:34369] regionserver.HRegionServer(3304): Received CLOSE for 2d66eb2b02a1587c2e82d321ca36ece9 2023-03-21 11:58:34,321 INFO [RS:0;jenkins-hbase20:34369] regionserver.HRegionServer(1145): stopping server jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:34,321 DEBUG [RS:0;jenkins-hbase20:34369] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6f18cc63 to 127.0.0.1:61761 2023-03-21 11:58:34,321 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 2d66eb2b02a1587c2e82d321ca36ece9, disabling compactions & flushes 2023-03-21 11:58:34,322 DEBUG [RS:0;jenkins-hbase20:34369] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-21 11:58:34,322 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1679399734440.2d66eb2b02a1587c2e82d321ca36ece9. 2023-03-21 11:58:34,322 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1679399734440.2d66eb2b02a1587c2e82d321ca36ece9. 2023-03-21 11:58:34,322 INFO [RS:0;jenkins-hbase20:34369] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-03-21 11:58:34,322 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1679399734440.2d66eb2b02a1587c2e82d321ca36ece9. after waiting 0 ms 2023-03-21 11:58:34,322 INFO [RS:0;jenkins-hbase20:34369] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-03-21 11:58:34,322 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1679399734440.2d66eb2b02a1587c2e82d321ca36ece9. 2023-03-21 11:58:34,322 INFO [RS:0;jenkins-hbase20:34369] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-03-21 11:58:34,322 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing 2d66eb2b02a1587c2e82d321ca36ece9 1/1 column families, dataSize=78 B heapSize=488 B 2023-03-21 11:58:34,322 INFO [RS:0;jenkins-hbase20:34369] regionserver.HRegionServer(3304): Received CLOSE for 1588230740 2023-03-21 11:58:34,322 INFO [RS:0;jenkins-hbase20:34369] regionserver.HRegionServer(1475): Waiting on 2 regions to close 2023-03-21 11:58:34,323 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-03-21 11:58:34,323 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-03-21 11:58:34,323 DEBUG [RS:0;jenkins-hbase20:34369] regionserver.HRegionServer(1479): Online Regions={2d66eb2b02a1587c2e82d321ca36ece9=hbase:namespace,,1679399734440.2d66eb2b02a1587c2e82d321ca36ece9., 1588230740=hbase:meta,,1.1588230740} 2023-03-21 11:58:34,323 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-03-21 11:58:34,323 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-03-21 11:58:34,323 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-03-21 11:58:34,323 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=20.94 KB heapSize=36.31 KB 2023-03-21 11:58:34,324 DEBUG [RS:0;jenkins-hbase20:34369] regionserver.HRegionServer(1505): Waiting on 1588230740, 2d66eb2b02a1587c2e82d321ca36ece9 2023-03-21 11:58:34,406 INFO [regionserver/jenkins-hbase20:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-03-21 11:58:34,510 INFO [regionserver/jenkins-hbase20:0.Chore.1] hbase.ScheduledChore(146): Chore: CompactionChecker was stopped 2023-03-21 11:58:34,510 INFO [regionserver/jenkins-hbase20:0.Chore.1] hbase.ScheduledChore(146): Chore: MemstoreFlusherChore was stopped 2023-03-21 11:58:34,525 DEBUG [RS:0;jenkins-hbase20:34369] regionserver.HRegionServer(1505): Waiting on 1588230740, 2d66eb2b02a1587c2e82d321ca36ece9 2023-03-21 11:58:34,725 DEBUG [RS:0;jenkins-hbase20:34369] regionserver.HRegionServer(1505): Waiting on 1588230740, 2d66eb2b02a1587c2e82d321ca36ece9 2023-03-21 11:58:34,733 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/hbase/namespace/2d66eb2b02a1587c2e82d321ca36ece9/.tmp/info/f63e0c1115654903a8f3c15bb4133694 2023-03-21 11:58:34,735 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=18.29 KB at sequenceid=93 (bloomFilter=false), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/hbase/meta/1588230740/.tmp/info/063c56b8ea574ba386a7bd956426bc86 2023-03-21 11:58:34,743 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/hbase/namespace/2d66eb2b02a1587c2e82d321ca36ece9/.tmp/info/f63e0c1115654903a8f3c15bb4133694 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/hbase/namespace/2d66eb2b02a1587c2e82d321ca36ece9/info/f63e0c1115654903a8f3c15bb4133694 2023-03-21 11:58:34,744 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 063c56b8ea574ba386a7bd956426bc86 2023-03-21 11:58:34,749 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/hbase/namespace/2d66eb2b02a1587c2e82d321ca36ece9/info/f63e0c1115654903a8f3c15bb4133694, entries=2, sequenceid=6, filesize=4.8 K 2023-03-21 11:58:34,750 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for 2d66eb2b02a1587c2e82d321ca36ece9 in 428ms, sequenceid=6, compaction requested=false 2023-03-21 11:58:34,755 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/hbase/namespace/2d66eb2b02a1587c2e82d321ca36ece9/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=1 2023-03-21 11:58:34,756 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1679399734440.2d66eb2b02a1587c2e82d321ca36ece9. 2023-03-21 11:58:34,756 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 2d66eb2b02a1587c2e82d321ca36ece9: 2023-03-21 11:58:34,756 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1679399734440.2d66eb2b02a1587c2e82d321ca36ece9. 2023-03-21 11:58:34,926 DEBUG [RS:0;jenkins-hbase20:34369] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-03-21 11:58:35,126 DEBUG [RS:0;jenkins-hbase20:34369] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-03-21 11:58:35,153 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=588 B at sequenceid=93 (bloomFilter=false), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/hbase/meta/1588230740/.tmp/rep_barrier/3cc46d87086041549db1bf09fb97f70c 2023-03-21 11:58:35,160 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 3cc46d87086041549db1bf09fb97f70c 2023-03-21 11:58:35,326 INFO [RS:0;jenkins-hbase20:34369] regionserver.HRegionServer(1475): Waiting on 1 regions to close 2023-03-21 11:58:35,326 DEBUG [RS:0;jenkins-hbase20:34369] regionserver.HRegionServer(1479): Online Regions={1588230740=hbase:meta,,1.1588230740} 2023-03-21 11:58:35,327 DEBUG [RS:0;jenkins-hbase20:34369] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-03-21 11:58:35,527 DEBUG [RS:0;jenkins-hbase20:34369] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-03-21 11:58:35,570 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.08 KB at sequenceid=93 (bloomFilter=false), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/hbase/meta/1588230740/.tmp/table/08cfb384659d4895a9c24edac0d0f284 2023-03-21 11:58:35,573 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 08cfb384659d4895a9c24edac0d0f284 2023-03-21 11:58:35,573 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/hbase/meta/1588230740/.tmp/info/063c56b8ea574ba386a7bd956426bc86 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/hbase/meta/1588230740/info/063c56b8ea574ba386a7bd956426bc86 2023-03-21 11:58:35,576 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 063c56b8ea574ba386a7bd956426bc86 2023-03-21 11:58:35,576 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/hbase/meta/1588230740/info/063c56b8ea574ba386a7bd956426bc86, entries=22, sequenceid=93, filesize=7.1 K 2023-03-21 11:58:35,577 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/hbase/meta/1588230740/.tmp/rep_barrier/3cc46d87086041549db1bf09fb97f70c as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/hbase/meta/1588230740/rep_barrier/3cc46d87086041549db1bf09fb97f70c 2023-03-21 11:58:35,580 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 3cc46d87086041549db1bf09fb97f70c 2023-03-21 11:58:35,580 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/hbase/meta/1588230740/rep_barrier/3cc46d87086041549db1bf09fb97f70c, entries=6, sequenceid=93, filesize=5.5 K 2023-03-21 11:58:35,580 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/hbase/meta/1588230740/.tmp/table/08cfb384659d4895a9c24edac0d0f284 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/hbase/meta/1588230740/table/08cfb384659d4895a9c24edac0d0f284 2023-03-21 11:58:35,583 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 08cfb384659d4895a9c24edac0d0f284 2023-03-21 11:58:35,583 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/hbase/meta/1588230740/table/08cfb384659d4895a9c24edac0d0f284, entries=9, sequenceid=93, filesize=5.5 K 2023-03-21 11:58:35,583 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.94 KB/21446, heapSize ~36.27 KB/37136, currentSize=0 B/0 for 1588230740 in 1260ms, sequenceid=93, compaction requested=false 2023-03-21 11:58:35,588 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/data/hbase/meta/1588230740/recovered.edits/96.seqid, newMaxSeqId=96, maxSeqId=1 2023-03-21 11:58:35,588 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-03-21 11:58:35,589 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-03-21 11:58:35,590 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-03-21 11:58:35,590 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-03-21 11:58:35,727 INFO [RS:0;jenkins-hbase20:34369] regionserver.HRegionServer(1171): stopping server jenkins-hbase20.apache.org,34369,1679399732030; all regions closed. 2023-03-21 11:58:35,734 DEBUG [RS:0;jenkins-hbase20:34369] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/oldWALs 2023-03-21 11:58:35,734 INFO [RS:0;jenkins-hbase20:34369] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase20.apache.org%2C34369%2C1679399732030.meta:.meta(num 1679399734213) 2023-03-21 11:58:35,737 DEBUG [RS:0;jenkins-hbase20:34369] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/oldWALs 2023-03-21 11:58:35,738 INFO [RS:0;jenkins-hbase20:34369] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase20.apache.org%2C34369%2C1679399732030:(num 1679399733641) 2023-03-21 11:58:35,738 DEBUG [RS:0;jenkins-hbase20:34369] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-21 11:58:35,738 INFO [RS:0;jenkins-hbase20:34369] regionserver.LeaseManager(133): Closed leases 2023-03-21 11:58:35,738 INFO [RS:0;jenkins-hbase20:34369] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase20:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-03-21 11:58:35,738 INFO [regionserver/jenkins-hbase20:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-03-21 11:58:35,739 INFO [RS:0;jenkins-hbase20:34369] ipc.NettyRpcServer(158): Stopping server on /148.251.75.209:34369 2023-03-21 11:58:35,745 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-03-21 11:58:35,745 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase20.apache.org,34369,1679399732030 2023-03-21 11:58:35,745 ERROR [Listener at localhost.localdomain/37475-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@539e0dd0 rejected from java.util.concurrent.ThreadPoolExecutor@57a1837b[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2404] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-03-21 11:58:35,745 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase20.apache.org,34369,1679399732030] 2023-03-21 11:58:35,746 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-03-21 11:58:35,747 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase20.apache.org,34369,1679399732030; numProcessing=1 2023-03-21 11:58:35,747 ERROR [Listener at localhost.localdomain/37475-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@a8ac04 rejected from java.util.concurrent.ThreadPoolExecutor@57a1837b[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2404] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-03-21 11:58:35,749 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase20.apache.org,34369,1679399732030 already deleted, retry=false 2023-03-21 11:58:35,749 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase20.apache.org,34369,1679399732030 expired; onlineServers=0 2023-03-21 11:58:35,749 INFO [RegionServerTracker-0] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase20.apache.org,34627,1679399731089' ***** 2023-03-21 11:58:35,749 INFO [RegionServerTracker-0] regionserver.HRegionServer(2310): STOPPED: Cluster shutdown set; onlineServer=0 2023-03-21 11:58:35,749 DEBUG [M:0;jenkins-hbase20:34627] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@79d82f7f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase20.apache.org/148.251.75.209:0 2023-03-21 11:58:35,749 INFO [M:0;jenkins-hbase20:34627] regionserver.HRegionServer(1145): stopping server jenkins-hbase20.apache.org,34627,1679399731089 2023-03-21 11:58:35,749 INFO [M:0;jenkins-hbase20:34627] regionserver.HRegionServer(1171): stopping server jenkins-hbase20.apache.org,34627,1679399731089; all regions closed. 2023-03-21 11:58:35,749 DEBUG [M:0;jenkins-hbase20:34627] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-21 11:58:35,749 DEBUG [M:0;jenkins-hbase20:34627] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-03-21 11:58:35,749 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-03-21 11:58:35,749 DEBUG [M:0;jenkins-hbase20:34627] cleaner.HFileCleaner(317): Stopping file delete threads 2023-03-21 11:58:35,749 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.small.0-1679399733347] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.small.0-1679399733347,5,FailOnTimeoutGroup] 2023-03-21 11:58:35,749 INFO [M:0;jenkins-hbase20:34627] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-03-21 11:58:35,749 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.large.0-1679399733342] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.large.0-1679399733342,5,FailOnTimeoutGroup] 2023-03-21 11:58:35,749 INFO [M:0;jenkins-hbase20:34627] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-03-21 11:58:35,750 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-03-21 11:58:35,750 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-21 11:58:35,750 INFO [M:0;jenkins-hbase20:34627] hbase.ChoreService(369): Chore service for: master/jenkins-hbase20:0 had [] on shutdown 2023-03-21 11:58:35,750 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-03-21 11:58:35,750 DEBUG [M:0;jenkins-hbase20:34627] master.HMaster(1512): Stopping service threads 2023-03-21 11:58:35,751 INFO [M:0;jenkins-hbase20:34627] procedure2.RemoteProcedureDispatcher(118): Stopping procedure remote dispatcher 2023-03-21 11:58:35,751 ERROR [M:0;jenkins-hbase20:34627] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-13,5,PEWorkerGroup] 2023-03-21 11:58:35,751 INFO [M:0;jenkins-hbase20:34627] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-03-21 11:58:35,751 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-03-21 11:58:35,751 DEBUG [M:0;jenkins-hbase20:34627] zookeeper.ZKUtil(398): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-03-21 11:58:35,752 WARN [M:0;jenkins-hbase20:34627] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-03-21 11:58:35,752 INFO [M:0;jenkins-hbase20:34627] assignment.AssignmentManager(315): Stopping assignment manager 2023-03-21 11:58:35,752 INFO [M:0;jenkins-hbase20:34627] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-03-21 11:58:35,752 DEBUG [M:0;jenkins-hbase20:34627] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-03-21 11:58:35,753 INFO [M:0;jenkins-hbase20:34627] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-21 11:58:35,753 DEBUG [M:0;jenkins-hbase20:34627] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-21 11:58:35,753 DEBUG [M:0;jenkins-hbase20:34627] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-03-21 11:58:35,753 DEBUG [M:0;jenkins-hbase20:34627] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-21 11:58:35,753 INFO [M:0;jenkins-hbase20:34627] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=345.35 KB heapSize=394.05 KB 2023-03-21 11:58:35,766 INFO [M:0;jenkins-hbase20:34627] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=345.35 KB at sequenceid=579 (bloomFilter=true), to=hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/c9766774863f497cacd029bfaa243253 2023-03-21 11:58:35,769 INFO [M:0;jenkins-hbase20:34627] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for c9766774863f497cacd029bfaa243253 2023-03-21 11:58:35,769 DEBUG [M:0;jenkins-hbase20:34627] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/c9766774863f497cacd029bfaa243253 as hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/c9766774863f497cacd029bfaa243253 2023-03-21 11:58:35,773 INFO [M:0;jenkins-hbase20:34627] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for c9766774863f497cacd029bfaa243253 2023-03-21 11:58:35,773 INFO [M:0;jenkins-hbase20:34627] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35467/user/jenkins/test-data/4d2c924e-1163-b24d-4510-fbb422e9f84c/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/c9766774863f497cacd029bfaa243253, entries=65, sequenceid=579, filesize=24.7 K 2023-03-21 11:58:35,774 INFO [M:0;jenkins-hbase20:34627] regionserver.HRegion(2948): Finished flush of dataSize ~345.35 KB/353640, heapSize ~394.04 KB/403496, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 21ms, sequenceid=579, compaction requested=false 2023-03-21 11:58:35,777 INFO [M:0;jenkins-hbase20:34627] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-21 11:58:35,778 DEBUG [M:0;jenkins-hbase20:34627] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-03-21 11:58:35,785 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-03-21 11:58:35,785 INFO [M:0;jenkins-hbase20:34627] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-03-21 11:58:35,786 INFO [M:0;jenkins-hbase20:34627] ipc.NettyRpcServer(158): Stopping server on /148.251.75.209:34627 2023-03-21 11:58:35,787 DEBUG [M:0;jenkins-hbase20:34627] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase20.apache.org,34627,1679399731089 already deleted, retry=false 2023-03-21 11:58:35,849 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-03-21 11:58:35,849 INFO [RS:0;jenkins-hbase20:34369] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase20.apache.org,34369,1679399732030; zookeeper connection closed. 2023-03-21 11:58:35,849 ERROR [Listener at localhost.localdomain/37475-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@781a2413 rejected from java.util.concurrent.ThreadPoolExecutor@57a1837b[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2404] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-03-21 11:58:35,850 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): regionserver:34369-0x10033bc54df0001, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-03-21 11:58:35,850 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@73835928] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@73835928 2023-03-21 11:58:35,850 ERROR [Listener at localhost.localdomain/37475-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@3701a27d rejected from java.util.concurrent.ThreadPoolExecutor@57a1837b[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2404] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-03-21 11:58:35,851 INFO [Listener at localhost.localdomain/37475] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 1 regionserver(s) complete 2023-03-21 11:58:35,949 INFO [M:0;jenkins-hbase20:34627] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase20.apache.org,34627,1679399731089; zookeeper connection closed. 2023-03-21 11:58:35,949 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-03-21 11:58:35,950 ERROR [Listener at localhost.localdomain/37475-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@116817d0 rejected from java.util.concurrent.ThreadPoolExecutor@720d5bf4[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4021] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-03-21 11:58:35,950 DEBUG [Listener at localhost.localdomain/37475-EventThread] zookeeper.ZKWatcher(600): master:34627-0x10033bc54df0000, quorum=127.0.0.1:61761, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-03-21 11:58:35,950 ERROR [Listener at localhost.localdomain/37475-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@20c45420 rejected from java.util.concurrent.ThreadPoolExecutor@720d5bf4[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4021] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-03-21 11:58:35,951 WARN [Listener at localhost.localdomain/37475] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-03-21 11:58:35,955 INFO [Listener at localhost.localdomain/37475] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-03-21 11:58:36,062 WARN [BP-1525689917-148.251.75.209-1679399728648 heartbeating to localhost.localdomain/127.0.0.1:35467] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-03-21 11:58:36,062 WARN [BP-1525689917-148.251.75.209-1679399728648 heartbeating to localhost.localdomain/127.0.0.1:35467] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1525689917-148.251.75.209-1679399728648 (Datanode Uuid 13b4a3a9-8383-46bc-9d4a-34a9f4509b76) service to localhost.localdomain/127.0.0.1:35467 2023-03-21 11:58:36,065 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/bb9d57e0-7c01-5166-0245-f9bc94a789fd/cluster_9b0830d2-29fd-b47c-c103-f4aba275458e/dfs/data/data1/current/BP-1525689917-148.251.75.209-1679399728648] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-03-21 11:58:36,066 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/bb9d57e0-7c01-5166-0245-f9bc94a789fd/cluster_9b0830d2-29fd-b47c-c103-f4aba275458e/dfs/data/data2/current/BP-1525689917-148.251.75.209-1679399728648] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-03-21 11:58:36,098 INFO [Listener at localhost.localdomain/37475] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost.localdomain:0 2023-03-21 11:58:36,252 INFO [Listener at localhost.localdomain/37475] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-03-21 11:58:36,415 INFO [Listener at localhost.localdomain/37475] hbase.HBaseTestingUtility(1293): Minicluster is down